Obstacle detection system and method therefor

ABSTRACT

An obstacle detection system using stereo cameras mounted on a vehicle, to detect an obstacle existing on a ground plane at a high speed and in a high precision even with the stereo cameras being uncalibrated and with a vibration during a traveling and a change in the inclination of the ground plane. The obstacle detection system comprises: a plurality of uncalibrated TV cameras for inputting stereo images; an image storage unit  2  for storing a plurality of images inputted from the TV cameras; a feature extraction unit  3  for extracting a plurality of mutually parallel lines existing on the ground plane; a parameter computation unit  4  for determining a relation to hold between the projected positions of an arbitrary point of the ground plane upon the individual images, from the plurality of lines extracted by the feature extraction unit  3 ; and a detection unit  5  for detecting an object having a height from the ground plane, by using the relation determined by the parameter computation unit  4.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. application Ser. No. 11/116,307filed on Apr. 28, 2005, which is a divisional of 09/659,815 filed onSep. 11, 2000, all of which claim priority to Japanese PatentApplication No. 11-255459 filed on Sep. 9, 1999, Japanese PatentApplication No. 11-272577 filed on Sep. 27, 1999, Japanese PatentApplication No. 2000-100784 filed on Apr. 3, 2000, and Japanese PatentApplication No. 2000-159177 filed on Mar. 31, 2000. The contents of eachof these documents are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Technical Field

The present invention relates to an obstacle detection system fordetecting an obstacle existing on a ground, such as a preceding vehicle,a pedestrian or a parked vehicle with cameras mounted on a vehicle, soas to support the safe drive of an automobile.

2. Background Art

The technology for detecting an obstacle with a sensor is coarselydivided into one utilizing a laser beam or ultrasonic waves and oneutilizing TV cameras. The technology utilizing the laser beam costs highand is impractical. On the other hand, the technology utilizing theultrasonic waves has a low resolution so that it is troubled by thedetection precision of the obstacle.

On the contrary, the TV cameras are relatively inexpensive so that theyare suitable for the obstacle detection from the aspects of theresolution, the measuring precision and the measuring range. In the caseof using the TV cameras, there are methods of employing one camera and aplurality of cameras (or stereo cameras).

In the method of employing the single camera, the ground region and theobstacle region are separated with the clue of the informations such asthe intensities, colors or textures of one image taken by the camera.For example, the intermediate intensity region of a low chroma, i.e., agray region is extracted from the image to determine the ground regionor a region of few textures so that the ground region is extracted whileleaving the remaining region as the obstacle region. However, there aremany obstacles having intensities, colors or textures similar to thoseof the ground. Therefore, this method finds it difficult to separate theobstacle region and the ground region under the general situations.

On the contrary, the method using a plurality of cameras detects theobstacle with a clue to three-dimensional informations. The technologyfor obtaining the three-dimensional informations of an object scene byusing the plurality of cameras is generally called the “stereo vision”.According to this stereo vision, given corresponding points betweenstereo images, it is possible to determine the three dimensionalposition. If the positions and orientations of the individual cameraswith respect to the ground plane are predetermined, the height of anarbitrary point in the images from the ground plane can be obtained bythe stereo vision. Depending upon the presence or absence of the height,therefore, it is possible to separate the obstacle region and the groundregion. It is difficult for the method using the single camera to detectthe region having intensities, colors and textures similar to those ofthe ground, as the obstacle. According to the stereo vision, however,the obstacle is detected with a clue to the height from the ground planeso that the obstacle detection can be made in a more general scene.

The ordinary stereovision is a technology for determining the distancesof an arbitrary point on the image from the stereo cameras. For thistechnology, it is necessary to determine parameters in advance on thespacing and directions of the plurality of cameras and the focal lengthsand the principal points of the camera lenses. The work for determiningthe parameters is called the “calibration”. For this calibration, thereare prepared a number of points, the three-dimensional locations ofwhich are known. The projected locations of the points on the images aredetermined to compute the parameters on the locations and positions ofthe cameras and the focal lengths of the camera lenses. However, theseoperations require a long time and many works to obstruct the practicalobstacle detection by the stereo vision.

If it is sufficient to separate the ground region and the obstacleregion on the images, however, the calibrations are not necessarilyrequired. If the projected points of a point of the ground plane on theleft and right images are designated by (u₁, v₁) and (u_(r), v_(r)), thefollowing relation holds:

$\begin{matrix}\begin{matrix}{{u_{r} = \frac{{h_{11}u_{l}} + {h_{12}v_{l}} + h_{13}}{{h_{31}u_{l}} + {h_{32}v_{l}} + h_{33}}},} \\{v_{r} = \frac{{h_{21}u_{l}} + {h_{22}v_{l}} + h_{23}}{{h_{31}u_{l}} + {h_{32}v_{l}} + h_{33}}}\end{matrix} & (1)\end{matrix}$

h=(h₁₁, h₁₂, h₁₃, h₂₁, h₂₂, h₂₃, h₃₁ h₃₂, h₃₃)^(T) (T designates atransposition symbol) are parameters depending upon the locations andpositions of the individual cameras with respect to the ground plane andupon the focal lengths and image origins of the lenses of the individualcameras. The parameters h are predetermined from the projected points offour or more points of the ground plane on the left and right images. Byusing these relations, the corresponding point P′ (u_(r), v_(r)) on theright image is determined when it is assumed that an arbitrary pointP(u_(l), v_(l)) on the left image is present on the ground plane.

If the point P is present on the ground plane, the points P and P′ are aset of the correct corresponding points so that the difference betweentheir intensities becomes small. When the points P and P′ have largelydifferent intensities, therefore, it is decided that the point P belongsto the obstacle region. In the following, Equation 1 will be called the“ground plane constraint”.

In this method, the search for corresponding points is also unnecessary.The ordinary stereo method requires the search for matching pointsbetween the left and right images so that its computation cost is highbecause the correspondence is made by the search computation. However,the aforementioned method requires no correspondence search so that thecomputational cost is extremely inexpensive.

If the stereo cameras are fixed in the three-dimensional space, theobstacle existing on the ground plane can be detected by the parametersH once determined. While the vehicle is running, however, the relativegeometric relationship between the ground plane and the individualcameras are changed time after time by the vibration of the vehicleitself and the change in the inclination of the ground. In short, theparameters h change during the traveling so that the ground planeconstraint determined at a still time cannot be used for the obstacledetections during the traveling.

In order to solve this problem, there has been usually used a method fordetecting the obstacle by computing the ground plane constraint using anumber of featuring points (e.g., the corner points of the paints on theground) on the ground plane. It is, however, difficult to extract thenumerous featuring points on the ground plane, and it frequently occursthat the featuring points on the obstacle are erroneously extracted.Moreover, the correspondence search of the extracted featuring pointshas to be performed to raise the computation cost. Also, with the largenumber of parameters to be determined, there has been a problem that itis seriously difficult to determine the ground plane constraint stably.

OBJECTS OF THE INVENTION

As described hereinbefore, the obstacle detection system is coarselydivided into one using the laser beam or the ultrasonic waves and oneusing the TV cameras. However, the obstacle detection system utilizingthe laser beam and the ultrasonic waves is troubled by its high price orby its low measurement precision.

On the other hand, the obstacle detection system utilizing the TVcameras is troubled by a limited using environment, by the necessity forthe calibration requiring the long time and the many works, by thenecessity for the correspondence search of the left and right images ofthe high computation cost, and by the absence of the practicalcounter-measures for the vibration during the traveling of the vehicleand the inclination of the ground.

Therefore, the present invention has been conceived in view of theaforementioned background and contemplates to provide an obstacledetection system and a method therefor, which is enabled to detect anobstacle existing on the ground plane at a high speed even with thevibration during a traveling and an inclination of the ground itself, bydetermining the geometric relations between the ground plane and theindividual cameras with less troubles of the calibration and using twolines segmenting the driving lane.

DISCLOSURE OF THE INVENTION

According to the invention of Claim 1, there is provided an obstacledetection system comprising:

a plurality of TV cameras for inputting an image;

an image storage unit for storing a plurality of images inputted fromthe TV cameras;

a feature extraction unit for extracting a line existing in a plane of athree-dimensional space, from the images;

a parameter computation unit for determining a relation to hold betweenthe projected positions of an arbitrary point in the plane upon theindividual images, from the line extracted by the feature extractionunit; and

a detection unit for detecting a region absent from the plane, by usingthe relation computed by the parameter computation unit.

According to the invention of Claim 2, there is provided an obstacledetection system according to Claim 1,

wherein the TV cameras are unknown on their relative positions andorientations and on their focal lengths and principal points.

According to the invention of Claim 3, there is provided an obstacledetection system according to Claim 1 or 2,

wherein the relation to hold between the projected points of anarbitrary point of the plane in the three-dimensional space upon theindividual images is expressed by a two-dimensional affinetransformation thereby to determine the affine transformationparameters.

According to the invention of Claim 4, there is provided an obstacledetection system according to any of Claims 1 to 3,

wherein the feature extraction unit extracts a plurality of lines, asexisting on the plane in the three-dimensional space and parallel toeach other in the three-dimensional space, from the images, anddetermines the vanishing points of the lines.

According to the invention of Claim 5, there is provided an obstacledetection system according to any of Claims 1 to 3,

wherein the feature extraction unit extracts a plurality of lines, asexisting on the plane in the three-dimensional space and parallel toeach other in the three-dimensional space, from the images, anddetermines the inclinations of the lines on the images and the vanishingpoints of the lines.

According to the invention of Claim 6, there is provided an obstacledetection system comprising:

a plurality of image pickup units having light receiving units disposedon a driver's own vehicle at a substantial spacing from each other fortaking the regions, to which the light receiving units are directed,simultaneously as images;

an image storage unit for storing the images taken by the image pickupunits;

a feature extraction unit for extracting such ones of the regions takenby the image pickup units as correspond to parallel members disposedgenerally in parallel with each other on a plane, as can be traveled bythe own vehicle, from the first image taken by the first image pickupunit and the second image taken by the second image pickup unit, asstored in the image storage unit, to determine a point of intersectionat which the extracted regions intersect in the first and second images;

a difference detection unit for determining the corresponding region inthe second image, as corresponding to an arbitrary region in the firstimage, assuming that the arbitrary region is caused by the plane, fromthe epipolar constraint to hold between the extracted region and thefirst and second images, to compare the intensities of the arbitraryregion and the corresponding region, thereby to extract the regionhaving a substantially different intensity as an obstacle region toobtain an obstacle region image from the extracted result; and

a height computation unit for extracting a polygonal region, as composedof an intensity higher than a standard value, of the obstacle regionimage thereby to detect as a true obstacle region the polygonal regionof a threshold or higher value of the ratio which is determined from thevertical size of the polygonal region in the obstacle region image andthe size from the lower end of the polygonal region to the scan-lineincluding the vanishing point for the lines on a plane.

According to the invention of Claim 7, there is provided an obstacledetection system comprising:

a plurality of image pickup units having light receiving units disposedat a substantial spacing from each other for taking the regions, towhich the light receiving units are directed, simultaneously as images;

an image storage unit for storing the images taken by the image pickupunits;

a difference detection unit for determining the corresponding region inthe second image, as corresponding to an arbitrary region in the firstimage, assuming that the arbitrary region is caused by a plane in athree-dimensional space, to compare the intensities of the arbitraryregion and the corresponding region, thereby to extract the regionhaving a substantially different intensity as an obstacle region toobtain an obstacle region image from the extracted result; and

a height computation unit for extracting a polygonal region, as composedof an intensity higher than a standard value, of the obstacle regionimage thereby to detect as a true obstacle region the polygonal regionof a threshold or higher value of the ratio which is determined from thevertical size of the polygonal region in the obstacle region image andthe size from the lower end of the polygonal region to the scan-line setin the obstacle region image.

According to the invention of Claim 8, there is provided an obstacledetection system comprising:

a first image pickup unit and a second image pickup unit for obtaining afirst image information of a first image and a second image informationof a second image, respectively, by taking the surrounding region of adriver's own vehicle substantially simultaneously as images formed of aset of pixels from light receiving units arranged at a spacing on theown vehicle;

an image information storage unit for storing the first imageinformation and the second image information;

an intensity difference image forming unit for forming an intensitydifference image by determining the corresponding pixels in the secondimage of the second image information, as assuming that an arbitrarypixel of the first image of the first image information stored in theimage information storage unit exists on the ground plane being traveledby the own vehicle, to determine the intensity difference between thearbitrary pixel and the corresponding pixel;

a discrimination image forming unit for obtaining a discrimination imageby discriminating each pixel in the intensity difference image into apixel having an intensity difference no less than a standard value and apixel having an intensity difference less than the standard value; and

a decision unit for detecting and deciding a region having a generallywedge-shaped set of pixels in the discrimination image as an obstacleregion.

According to the invention of Claim 9, there is provided an obstacledetection system according to Claim 8,

wherein the detection unit decides that the lowermost pixel in thewedge-shaped region of the discrimination image is either at a point ofcontact between the obstacle region in the first or second image, ascorresponding to the discrimination image, and the ground being traveledby the own vehicle, or at a portion of the obstacle region and theclosest to the own vehicle.

According to the invention of Claim 10, there is provided an obstacledetection system according to Claim 8 or 9,

wherein the detection unit decides that such one of the generallywedge-shaped region existing generally in the scan-line direction of thefirst and second images, as corresponding to the discrimination image,that its side is at a higher location of the first and second imagesthan the apexes opposed thereto, is the obstacle region.

According to the invention of Claim 11, there is provided an obstacledetection system according to Claim 8,

wherein the detection unit decides one pair of wedge-shaped regionsgenerally of the same shape, as located at a spacing on the generallyidentical scan-line in the discrimination image, and decides the regionbetween the paired wedge-shaped regions as the obstacle.

According to the invention of Claim 12, there is provided an obstacledetection system comprising:

an image input unit for inputting and storing at least two images ofdifferent pickup points;

a feature extraction unit for extracting a projected point of a motionof an object, as stands still or moves on a plane in a three-dimensionalspace with respect to the pickup point of a standard image, upon thestandard image corresponding to an infinite point, by employing one ofthe images stored by the image input unit as the standard image and theother as a reference image;

a detection unit for calculating a corresponding point on the referenceimage when it is assumed that an arbitrary point on the standard imageis on the plane, to detect a point non-existing on the plane from theintensity difference between the arbitrary point and the correspondingpoint; and

a contact time computing unit for computing the time period for thepoint non-existing on the plane to come to the taken point of thestandard image, on the basis of the point non-existing on the plane inthe standard image detected by the detection unit and the projectedpoint extracted from the feature extraction unit.

According to the invention of Claim 13, there is provided an obstacledetection system according to Claim 12,

wherein the contact time computation unit computes the time period forthe point on the boundary with the plane to come to the taken point ofthe standard image, on the basis of such ones of the points detected bythe detection unit but not existing on the plane, as are located on theboundary line with the plane in the standard image and are extracted asthe projected point by the feature extraction unit.

According to the invention of Claim 14, there is provided an obstacledetection system according to Claim 12 or 13,

wherein the feature extraction unit extracts a plurality of linesaligned to the direction of the motion of an object which stands stillor moves on the plane relative to the pickup point of the standardimage, to employ the point of intersection of the extracted lines, asthe projected point.

According to the invention of Claim 15, there is provided an obstacledetection system according to any Claims 12 to 14,

wherein the parameters are unknown including camera positions,orientations, the focal lengths and principal points.

According to the invention of Claim 16, there is provided an obstacledetection system according to any of Claims 12 to 15,

wherein the detection unit computes the corresponding point on thereference image when the arbitrary point on the standard image isassumed to be on the plane, to detect the non-existing point from asimilarity of the surrounding intensities of the arbitrary point and thecorresponding point.

According to the invention of Claim 17, there is provided an obstacledetection method comprising:

an image storage step of storing a plurality of images inputted from aplurality of TV cameras;

a feature extraction step of extracting a line existing in a plane of athree-dimensional space, from the images;

a parameter computation step of determining a relation to hold betweenthe projected positions of an arbitrary point in the plane upon theindividual images, from the line extracted at the feature extractionstep; and

a detection step of detecting a region absent from the plane, by usingthe relation computed at the parameter computation step.

According to the invention of Claim 18, there is provided an obstacledetection method according to Claim 17, wherein the TV cameras areunknown on their relative locations and positions and on their focallengths and principal points.

According to the invention of Claim 19, there is provided an obstacledetection method according to Claim 17 or 18,

wherein the parameter computation step expresses the relation to holdbetween the projected points of an arbitrary point of the plane in thethree-dimensional space upon the individual images by a two-dimensionalaffine transformation thereby to determine the affine transformationparameters.

According to the invention of Claim 20, there is provided an obstacledetection method according to any of Claims 17 to 19,

wherein the feature extraction step extracts a plurality of lines, asexisting on the plane in the three-dimensional space and parallel toeach other in the three-dimensional space, from the images, anddetermines the vanishing points of the lines.

According to the invention of Claim 21, there is provided an obstacledetection method according to any of Claims 17 to 19,

wherein the feature extraction step extracts a plurality of lines, asexisting on the plane in the three-dimensional space and parallel toeach other in the three-dimensional space, from the images, anddetermines the inclinations of the lines on the images and the vanishingpoints of the lines.

According to the invention of Claim 22, there is provided an obstacledetection method comprising:

a plurality of image pickup steps of taking the regions, to which lightreceiving units disposed on a driver's own vehicle at a substantialspacing from each other are directed, simultaneously as images;

an image storage step of storing the images taken by the image pickupsteps;

a feature extraction step of extracting such ones of the regions takenby the image pickup steps as correspond to parallel members disposedgenerally in parallel with each other on a plane, as can be traveled bythe own vehicle, from the first image taken by the first image pickupstep and the second image taken by the second image pickup step, asstored in the image storage step, to determine a point of intersectionat which the extracted regions intersect in the first and second images;

a difference detection step of determining the corresponding region inthe second image, as corresponding to an arbitrary region in the firstimage, assuming that the arbitrary region is caused by the plane, fromthe epipolar constraint to hold between the extracted region and thefirst and second images, to compare the intensities of the arbitraryregion and the corresponding region, thereby to extract the regionhaving a substantially different intensity as an obstacle region toobtain an obstacle region image from the extracted result; and

a height computation step of extracting a polygonal region, as composedof an intensity higher than a standard value, of the obstacle regionimage thereby to detect as a true obstacle region the polygonal regionof a threshold or higher value of the ratio which is determined from thevertical size of the polygonal region in the obstacle region image andthe size from the lower end of the polygonal region to the scan-line inthe obstacle region image including the intersection point.

According to the invention of Claim 23, there is provided an obstacledetection method comprising:

a plurality of image pickup steps of taking the regions, to which thelight receiving units disposed at a substantial spacing from each otherare directed, simultaneously as images;

an image storage step of storing the images taken by the image pickupsteps;

a difference detection step of determining the corresponding region inthe second image, as corresponding to an arbitrary region in the firstimage, assuming that the arbitrary region is caused by a plane in athree-dimensional space, to compare the intensities of the arbitraryregion and the corresponding region, thereby to extract the regionhaving a substantially different intensity as an obstacle region toobtain an obstacle region image from the extracted result; and

a height computation step of extracting a polygonal region, as composedof an intensity higher than a standard value, of the obstacle regionimage thereby to detect as a true obstacle region the polygonal regionof a threshold or higher value of the ratio which is determined from thevertical size of the polygonal region in the obstacle region image andthe size from the lower end of the polygonal region to the scan-line setin the obstacle region image.

According to the invention of Claim 24, there is provided an obstacledetection method comprising:

a first image pickup step and a second image pickup step of obtaining afirst image information of a first image and a second image informationof a second image, respectively, by taking the surrounding region of adriver's own vehicle substantially simultaneously as images formed of aset of pixels from light receiving units arranged at a spacing on theown vehicle;

an image information storage step of storing the first image informationand the second image information;

an intensity difference image forming step of forming an intensitydifference image by determining the corresponding pixels in the secondimage of the second image information, as assuming that an arbitrarypixel of the first image of the first image information stored in theimage information storage step exists on the ground plane being traveledby the own vehicle, to determine the intensity difference between thearbitrary pixel and the corresponding pixel;

a discrimination image forming step of obtaining a discrimination imageby discriminating each pixel in the intensity difference image into apixel having an intensity difference no less than a standard value and apixel having an intensity difference less than the standard value; and

a decision step of detecting and deciding a region having a generallywedge-shaped set of pixels in the discrimination image as an obstacleregion.

According to the invention of Claim 25, there is provided an obstacledetection method according to Claim 24,

wherein the detection step decides that the lowermost pixel in thewedge-shaped region of the discrimination image is either at a point ofcontact between the obstacle region in the first or second image, ascorresponding to the discrimination image, and the ground being traveledby the own vehicle, or at a portion of the obstacle region and theclosest to the own vehicle.

According to the invention of Claim 26, there is provided an obstacledetection method according to Claim 24 or 25,

wherein the detection step decides that such one of the generallywedge-shaped region existing generally in the scan-line direction of thefirst and second images, as corresponding to the discrimination image,that its side is at a higher location of the first and second imagesthan the apexes opposed thereto, is the obstacle region.

According to the invention of Claim 27, there is provided an obstacledetection method according to Claim 24,

wherein the detection step decides one pair of wedge-shaped regionsgenerally of the same shape, as located at a spacing on the generallyidentical scan-line in the discrimination image, and decides the regionbetween the paired wedge-shaped regions as the obstacle.

According to the invention of Claim 28, there is provided an obstacledetection method comprising:

an image input step of inputting and storing at least two images ofdifferent pickup points;

a feature extraction step of extracting a projected point of a motion ofan object, as stands still or moves on a plane in a three-dimensionalspace with respect to the pickup point of a standard image, upon thestandard image corresponding to an infinite point, by employing one ofthe images stored by the image input step as the standard image and theother as a reference image;

a detection step of calculating a corresponding point on the referenceimage when it is assumed that an arbitrary point on the standard imageis on the plane, to detect a point non-existing on the plane from theintensity difference between the arbitrary point and the correspondingpoint; and

a contact time computation step of computing the time period for thepoint non-existing on the plane to come to the taken point of thestandard image, on the basis of the point non-existing on the plane inthe standard image detected by the detection step and the projectedpoint extracted from the feature extraction step.

According to the invention of Claim 29, there is provided an obstacledetection method according to Claim 28,

wherein the contact time computation step computes the time period forthe point on the boundary with the plane to come to the taken point ofthe standard image, on the basis of such ones of the points detected bythe detection step but not existing on the plane, as are located on theboundary line with the plane in the standard image and are extracted asthe projected point by the feature extraction step.

According to the invention of Claim 30, there is provided an obstacledetection method according to Claim 28 or 29,

wherein the feature extraction step extracts a plurality of linesaligned to the direction of the motion of an object which stands stillor moves on the plane relative to the pickup point of the standardimage, to employ the point of intersection of the extracted lines, asthe projected point.

According to the invention of Claim 31, there is provided an obstacledetection method according to any Claims 28 to 30,

wherein the images are unknown on the relative locations and positionsof their pickup points and on their focal lengths and principal points.

According to the invention of Claim 32, there is provided an obstacledetection method according to any of Claims 28 to 31,

wherein the detection step computes the corresponding point on thereference image when the arbitrary point on the standard image isassumed to be on the plane, to detect the non-existing point from asimilarity of the surrounding intensities of the arbitrary point and thecorresponding point.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example of Embodiment 1;

FIG. 2 shows an entire construction of Embodiment 1;

FIG. 3 is a diagram for explaining a stereo camera coordinate system;

FIG. 4 is a diagram for explaining an inclined plane;

FIG. 5 is a diagram for explaining a parameter P;

FIG. 6 is a diagram for explaining two white lines on the two ends of aground and their vanishing point;

FIG. 7 is a diagram for explaining changes in the two white lines of thetwo ends of the ground being traveled;

FIG. 8 shows a modification of a detection unit 5;

FIG. 9 shows stereo images;

FIG. 10 is a diagram for explaining a right image and its transformedimage;

FIG. 11 is a diagram for explaining a left image and the transformedimage of the right image;

FIG. 12 is a diagram for explaining a positional relation between aground plane and a stereo camera;

FIG. 13 is a diagram for explaining an epipolar constraint of anobstacle detection system of the invention;

FIG. 14 is a diagram for explaining the actions of a feature extractionunit;

FIG. 15 is a diagram for explaining the actions of a differencedetection unit;

FIG. 16 is another diagram for explaining the actions of the differencedetection unit;

FIG. 17 is a block diagram of Embodiment 2-1 of the obstacle detectionsystem;

FIG. 18 is another diagram for explaining the actions of a heightdetection unit of Embodiment 2-1 of the obstacle detection system;

FIG. 19 is a diagram for explaining other actions of the heightdetection unit of Embodiment 2-1 of the obstacle detection system;

FIG. 20 is a block diagram of Embodiment 2-2 of the obstacle detectionsystem;

FIG. 21 is a diagram for explaining the actions of a differencedetection unit of Embodiment 2-2 of the obstacle detection system;

FIG. 22 is a diagram for explaining the difference detection unit ofEmbodiment 2-2 of the obstacle detection system;

FIG. 23 is a diagram for explaining the difference detection unit ofEmbodiment 2-2 of the obstacle detection system;

FIG. 24 is a diagram for explaining a positional relation between imagepickup means of the obstacle detection system of the invention and theground plane;

FIG. 25 is a diagram for explaining the epipolar constraint of theobstacle detection system of the invention;

FIG. 26 is a diagram for explaining the epipolar constraint of theobstacle detection system of the invention;

FIG. 27 is a diagram for explaining the epipolar constraint of theobstacle detection system of the invention;

FIG. 28 is a diagram for explaining the actions of an imagetransformation unit of the obstacle detection system of the invention;

FIG. 29 is a block diagram of Embodiment 3-1 of the obstacle detectionsystem of the invention;

FIG. 30 is a construction diagram of image pickup means of Embodiment3-1 of the obstacle detection system of the invention;

FIG. 31 is a diagram for explaining the actions of a detection unit ofEmbodiment 3-1 of the obstacle detection system of the invention;

FIG. 32 is a diagram for explaining a decision unit of Embodiment 3-1 ofthe obstacle detection system of the invention;

FIG. 33 is a diagram for explaining of the decision unit of Embodiment3-1 of the obstacle detection system of the invention;

FIG. 34 is a block diagram of Embodiment 3-2 of the obstacle detectionsystem of the invention;

FIG. 35 is a diagram for explaining the actions of a detection unit ofEmbodiment 3-2 of the obstacle detection system of the invention;

FIG. 36 is a diagram for explaining the actions of a detection unit ofEmbodiment 3-2 of the obstacle detection system of the invention;

FIG. 37 is a diagram for explaining the actions of a detection unit ofEmbodiment 3-2 of the obstacle detection system of the invention;

FIG. 38 is a diagram for explaining a corresponding search ofstereoscopic views;

FIG. 39 is a diagram for explaining of a driver's own vehicle ofEmbodiment 4;

FIG. 40 is a block diagram of the obstacle detection system ofEmbodiment 4;

FIG. 41 is a diagram for explaining the own coordinate system;

FIG. 42 is a diagram for explaining a linear extraction processing inthe feature extraction unit 3;

FIG. 43 is a diagram for explaining an obstacle detection processing inthe detection unit 4;

FIG. 44 is a diagram for explaining a depth;

FIG. 45 is a diagram (1) for explaining a time to contact;

FIG. 46 is a diagram (2) for explaining the time to contact;

FIG. 47 is a block diagram of a modification of the detection unit 4;

FIG. 48 shows stereo images;

FIG. 49 is a diagram for explaining a right image and its transformedimage; and

FIG. 50 is a diagram for explaining a left image and the transformedimage of a right image.

BEST MODES FOR CARRYING OUT THE INVENTION Embodiment 1

Embodiment 1 of the invention will be described with reference to theaccompanying drawings.

In this Embodiment, there is imagined a situation that an obstacleexisting on a ground plane such as a pedestrian, a preceding vehicle ora parked vehicle is detected by two left and right stereo camerasmounted on a vehicle, as shown in FIG. 1.

FIG. 2 shows a schematic construction of the obstacle detection systemaccording to this Embodiment. This obstacle detection system isconstructed to include an image input unit 1, an image storage unit 2, afeature extraction unit 3, a parameter computation unit 4 and adetection unit 5. The obstacle detection system predetermines, in astill state, a relation (as will be called the “ground planeconstraint”) to hold between the projected positions of a point of aground plane upon left and right images, and computes the ground planeconstraint at a traveling time, as changed time after time by thevibration of the driven vehicle and the inclination of the ground,exclusively from the motions of two white lines existing on the ground,thereby to discriminate an obstacle existing on the ground plane and theground region.

In this Embodiment, as shown in FIG. 3, there is set the stereo cameracoordinate system in which two white lines on the two ground ends aredesignated by 11 and 12, in which a Y-axis is taken in the direction ofthe straight lines 11 and 12 where as an X-axis and a Z-axis are takenin the transverse and vertical directions, respectively, and in which anX-Y plane is taken in a flat plane (or a standard plane) having noinclination. The image input unit 1 inputs two images by using two leftand right TV cameras. These two cameras need not be predetermined intheir locations and positions, but it is assumed that the individualcameras are fixed on the vehicle and not changed during the traveling.

The image storage unit 2 stores an image memory with the two imagesinputted from the image input unit 1.

The feature extraction unit 3 detects the two straight lines 11 and 12,as shown in FIG. 6, on the two images stored by the image storage unit 2and determines their point of intersection (or vanishing point). Thisstraight line detection is performed by using the edge extractionprocessing and the Hough transformation.

The parameter computation unit 4 computes the ground plane constraint atthe traveling time from both the ground plane constraint with respect tothe standard plane determined at the still time and the two white linesand their vanishing point determined by the feature extraction unit 3.Here will be described this method. If the projected point of a point(X, Y, Z) in a three-dimensional space is designated by (u, v), thefollowing relation generally holds:

$\begin{matrix}{u = \frac{{h_{11}X} + {h_{12}Y} + {h_{13}Z} + t_{1}}{{h_{31}X} + {h_{32}Y} + {h_{33}Z} + t_{3}}} & (2) \\{v = \frac{{h_{21}X} + {h_{22}Y} + {h_{23}Z} + t_{2}}{{h_{31}X} + {h_{32}Y} + {h_{33}Z} + t_{3}}} & (3)\end{matrix}$

wherein h=(h1₁, h1₂, . . . , and t₃)^(T) is a parameter relating to thelocations, the positions, the focal lengths and the principal points ofthe cameras. Since this parameter h indicates the same camera model,even if multiplied by a constant, the generality is not lost even if anarbitrary element of the parameter h is assumed to be “1”. Therefore,h₃₂ will be set to h₃₂=1.

In the stereo camera coordinate system shown in FIG. 3, the ground plane(or standard plane) is expressed by Z=0 so that the projected point ofthe point P(X, Y) on the ground plane is expressed by substituting Z=0into the foregoing Formula:

$\begin{matrix}\begin{matrix}{{u = \frac{{h_{11}X} + {h_{12}Y} + t_{1}}{{h_{31}X} + Y + t_{3}}},} \\{v = \frac{{h_{21}X} + {h_{22}Y} + t_{2}}{{h_{31}X} + Y + t_{3}}}\end{matrix} & (4)\end{matrix}$

Here will be considered the camera model under the following premises:

(a) The target area is relatively distant from the stereo cameras;(b) The longitudinal difference between the left and right camerapositions.

Under these premises, the following Formula holds:

[Denominator of formula (4)]

Y+β+h ₃₁ X+Δt ₃ ≅Y+β  (5)

wherein letter β indicates a displacement, as shown in FIG. 5, in theY-direction between the middle point of the view points of the left andright cameras and the origin of the coordinate, and t₃=β+Δt₃.

Therefore, Equation (4) can be simplified, as follows:

$\begin{matrix}{{{u \simeq \frac{{h_{11}X} + {h_{12}Y} + t_{1}}{Y + \beta}},{v \simeq \frac{{h_{21}X} + {h_{22}Y} + t_{2}}{Y + \beta}}}{{{{If}\mspace{14mu} {Yc}} = {Y + \beta}},}} & (6) \\{\begin{bmatrix}u \\v\end{bmatrix} = {{\begin{bmatrix}h_{11} & {t_{1} - {\beta \; h_{12}}} \\h_{21} & {t_{2} - {\beta \; h_{22}}}\end{bmatrix}\begin{bmatrix}{X/Y_{C}} \\{1/Y_{C}}\end{bmatrix}} + \begin{bmatrix}h_{12} \\h_{22}\end{bmatrix}}} & (7)\end{matrix}$

The matrix of the right side is designated by M. If the point ofintersection (or vanishing point) of the white lines 11 and 12 isexpressed by t=(u₀, v₀)^(T), (h₁₂, h₂₂)^(T)=t. If X=(X/Yc, 1/Yc)^(T) andif the projected points of the point P of the ground plane on the leftand right images are designated by u₁ and u_(r), the following Formulais obtained (t₁ and t_(r) are the vanishing point of the white lines):

u _(l) −t _(l) =M _(l) X, u _(r) −t _(r) =M _(r) X  (8)

Therefore, the following Formula is obtained:

u _(r) −t _(r) =M _(r) M _(l) ⁻¹(u _(l) −t _(l))=A(u _(l) −t _(l))  (9)

Here, letters “l” and “r” designate suffixes to the left and rightimages, respectively. Since the stereo cameras are not calibrated,letters M_(l) and M_(r) have unknown values, but letter A ispredetermined from the featuring point on the ground plane having noinclination at the still time.

It is assumed (FIG. 4) that the ground plane is changed from thestandard plane Z=0 to the inclined plane Z=pY by the change in theinclination of the ground and the vibration of the own vehicle at thetraveling time. The inclination in the X-direction can be generallyignored because it is sufficiently smaller than that in the Y-direction.If the line of intersection between the inclined plane and the standardplane is taken on the X-axis, the Formula of the inclined plane can beexpressed by Z=pY. Here will be described a method of computing theground plane constraint with respect to Z=pY from the motions (FIG. 7)of the two white lines. The projected positions (u′, v′) of the point(X, Y, Z) on the inclined plane on the images are expressed under theaforementioned two premises by substituting Z=pY into Equation (2), asfollows:

$\begin{matrix}{u^{\prime} = \frac{{h_{11}X} + {\left( {h_{12} + {p\; h_{13}}} \right)Y} + t_{1}}{{\left( {1 + {p\; h_{33}}} \right)Y} + \beta}} & (10)\end{matrix}$

If it is assumed that the inclination is small, namely, that p issubstantially equal to 0, the following Formula is obtained:

$\begin{matrix}{u^{\prime} \simeq \frac{{h_{11}X} + {\left( {h_{12} + {p\; h_{13}}} \right)Y} + t_{1}}{Y + \beta}} & (11)\end{matrix}$

If Yc=Y+β and if a similar transformation is made for v′ by Equation(3), the following Formula is obtained:

$\begin{matrix}{\begin{bmatrix}u^{\prime} \\v^{\prime}\end{bmatrix} = {{\begin{bmatrix}h_{11} & {t_{1} - {\beta \; u_{0}^{\prime}}} \\h_{21} & {t_{2} - {\beta \; v_{0}^{\prime}}}\end{bmatrix}\begin{bmatrix}{X/Y_{C}} \\{1/Y_{C}}\end{bmatrix}} + \begin{bmatrix}u_{0}^{\prime} \\v_{0}^{\prime}\end{bmatrix}}} & (12)\end{matrix}$

wherein (u₀′, v₀′)^(T)=t′ indicates the vanishing point of the two whitelines.

If:

Δu=(Δu,Δv)^(T) =u−t;

and

Δu′=(Δu′,Δv′)^(T) =u′−t′,

the foregoing Formula is transformed into because Δu=MX from Equation(7):

$\begin{matrix}\begin{matrix}{{\Delta \; u^{\prime}} = {{\begin{bmatrix}h_{11} & {t_{1} - {\beta \; u_{0}}} \\h_{21} & {t_{2} - {\beta \; v_{0}}}\end{bmatrix}X} + {{\beta \begin{bmatrix}0 & {\Delta \; u_{0}} \\0 & {\Delta \; v_{0}}\end{bmatrix}}X}}} \\{= {{\Delta \; u} + {{\beta/Y_{C}}\Delta \; t}}}\end{matrix} & (13)\end{matrix}$

wherein

Δt=(Δu ₀ ,Δv ₀)^(T) =t−t′.

From the Formula (7), the following Formula is obtained:

X=M⁻¹Δu  (14)

and if the following Formula is defined, 1/Yc=m₂₁Δu+m₂₂Δv:

$\begin{matrix}{M^{- 1} = \begin{bmatrix}m_{11} & m_{12} \\m_{21} & m_{22}\end{bmatrix}} & (15)\end{matrix}$

Therefore, Equation (13) is transformed into:

$\begin{matrix}{{\Delta \; u^{\prime}} = {\begin{bmatrix}{1 + {\beta_{1}\Delta \; u_{0}}} & {\beta_{2}\Delta \; u_{0}} \\{\beta_{1}\Delta \; v_{0}} & {1 + {\beta_{2}\Delta \; v_{0}}}\end{bmatrix}\Delta \; u}} & (16)\end{matrix}$

wherein it is assumed that β₁=m₂₁β and that β₂=m₂₂β. If one white lineon the images is changed, as shown in FIG. 7, by the inclination froml₁:Δv=p₁Δu to l₁′:Δv′=p₁′Δu′, the following Formula is obtained fromEquation (16):

(p ₁ ′Δu ₀ −Δv ₀)β₁ +p ₁(p ₁ ′Δu ₀ −Δv ₀)β₂ =p ₁ −p ₁′  (17)

If a similar transformation is made for the other white line (l₂→l₂′),the following Formula is obtained, and there are obtained twoone-dimensional equations on β=(β₁, β₂)^(T):

(p ₂ ′Δu ₀ −Δv ₀)β₁ +p ₁(p ₂ ′Δu ₀ −Δv ₀)β₂ =p ₂ −p ₂′  (18)

If the value β is determined from them, a matrix K of Equation (16) canbe determined. If the aforementioned processing is made for each of theleft and right images, the projected positions of the point on theground plane are transformed with the change in the inclination into:Δu₁′=K₁Δu₁ and Δu_(r)′=K_(r)Δu_(r).

By using Equation (9), therefore, the following Formula is obtained:

Δu_(r)′=K_(r)Δu_(r)=K_(r)AΔu_(l)=K_(r)AK_(l) ⁻¹Δu_(l)′  (19)

The value A of Equation (9) has been changed to A′=KrAKl⁻¹ by theinclination. Equation (19) expresses the ground plane constraint withrespect to the inclined plane.

The detection unit 5 detects the obstacle by using the ground planeconstraint determined by the parameter computation unit 4. Thecorresponding point (u_(r), v_(r)) on the right image of the case inwhich it is assumed that the brightness of an arbitrary point (u_(l),v_(l)) of the left image has an intensity I_(L)(u_(l), v_(l)) and thatthe point (u, v) exists on the ground plane is determined from Equation(19) to have an intensity I_(R)(u_(r), v_(r)). If the point (u, v) isactually present on the ground plane, points P and P′ are a set ofcorrect corresponding points so that they basically have an equalintensity.

In short, it is decided that the point P for D of the following Formulais not 0 or for D>Thr (Thr: a preset threshold value) belongs to theobstacle region:

D=|I _(L)(u _(l) ,v _(l))−I _(R)(u _(r) ,v _(r))|(|·|:Absolutevalue)  (20)

In these ways, the obstacle on the ground plane can be detected at thetraveling time from the stereo cameras mounted on the vehicle.

<Modification 1-1>

In Embodiment 1, the image input unit 1 inputs two images by arrangingthe two TV cameras transversely, which may be vertically arranged.

On the other hand, there may be arranged three or more cameras.

<Modification 1-2>

The feature extraction unit 3 has been described on the case in whichthe two lines on the ground plane are to be extracted, but three or morelines may be extracted.

<Modification 1-3>

When it is sufficient to consider only the vibration of the travelingown vehicle, β=0 (or β₁=β₂=0) may be assumed in Equation (16). Thematrix of the righthand side of Equation (16) is K=I (I: the unitmatrix), and A′=A in Equation (19). As a result, it is possible toobtain the ground plane constraint more quickly with respect to theinclined plane.

<Modification 1-4>

The detection unit 5 can take a construction shown in FIG. 8. Here, theconstruction is made of an image transformation unit 5-1 and adifference computation unit 5-2. The image transformation unit 5-1transforms a right image in accordance with the following procedure.Generally, the image can be expressed as a function ƒ(u, v) which usesthe point (u, v) on the image as a variable and which has an intensitydefined for each point. In the following, the image is thus expressed.

It is assumed that a stereo image shown in FIG. 9 is inputted and thatthe right image is expressed by g(u, v) and the transformed image isexpressed by g′ (u, v). This image g′ (u, v) is determined in thefollowing manner.

g′(u,v)=g(u′,v′)  (21)

wherein (u′, v′) is determined from Equation 19.

Here, the image g′ (u, v) is one which is obtained by the left camerawhen it is assumed that an arbitrary point on the image g(u, v) existson the ground plane.

From the right image of FIG. 10, for example, there is obtained thetransformed image, as shown in FIG. 10. As shown in FIG. 11, theprojected points of a point on the ground plane are identical betweenthe left image and the transformed image, but a point non-existing onthe ground plane, i.e., a point on the obstacle (i.e., the precedingvehicle in this case) is projected at a location difference according tothe height from the ground. Therefore, the obstacle on the ground planeis detected by taking a difference between the left image and thetransformed image. Specifically, the left image is expressed by f(u, v),and it is decided that the point P for D′ of the following Formula isnot 0 or for D′>Thr (Thr: a preset threshold value) belongs to theobstacle region:

D′=|f(u,v)−g′(u,v)|(|·|:Absolute value)  (22)

<Modification 1-5>

The detection unit 5 detects the difference between the two images bytaking the pixel difference, but may detect the difference by setting awindow of (2w+1)×(2w+1) for each point and by computing a normalizedcross correlation C of the intensities in the window. The correlation Cof the point (u, v) of two images F(u, v) and G(u, v) is expressed bythe following

Formula:

$\begin{matrix}{C = {\frac{1}{N}{\sum\limits_{\eta = {- \omega}}^{\omega}{\sum\limits_{\xi = {- \omega}}^{\omega}\frac{\left( {{F\left( {{u + \xi},{v + \eta}} \right)} - a_{1}} \right)\left( {{G\left( {{u + \xi},{v + \eta}} \right)} - a_{2}} \right)}{\sigma_{1}\sigma_{2}}}}}} & (23)\end{matrix}$

wherein: N=(2w+1)×(2w+1); a1 and a2 designate the averages of theintensities in the window of the two images; and σ₁ ² and σ₂ ² designatethe variances in the intensities in the window of the two images. Inthis case, it is decided that the point (u, v) for C<Thr (Thr: a presentthreshold value) belongs to the obstacle region.

<Modification 1-6>

In Embodiment 1, on the other hand, the two white lines on the twoground ends are extracted as the straight lines, but the white lines arecurves when the ground is curved. In this case, the obstacle can belikewise detected if the white lines are extracted as the curves.

<Modification 1-7>

The description has been made assuming that the ground plane is flat,but the obstacle can be detected as in the case of the flat plane evenfor the curved plane.

<Modification 1-8>

Although Embodiment 1 has been described on the obstacle detection fromthe cameras mounted on the vehicle, it could be applied to theautonomous run of a moving robot, for example, but this method shouldnot be limited to the obstacle detection from the vehicle-mountedcameras.

Embodiment 2

Here, the corresponding point search is not necessary if it issufficient to separate the ground region and the obstacle region on theimages, but the height from the ground plane can be discriminated by thefollowing manner, for example.

Here, the numerals in Equations are independent of those of Embodiment 1and are newly started from (1).

If the projected points of a point of the ground plane on the left andright images are designated by (u, v) and (u′, v′), respectively, thefollowing relation holds:

$\begin{matrix}{{u^{\prime} = \frac{{h_{11}u} + {h_{12}v} + h_{13}}{{h_{31}u} + {h_{32}v} + h_{33}}},{v^{\prime} = \frac{{h_{21}u} + {h_{22}v} + h_{23}}{{h_{31}u} + {h_{32}v} + h_{33}}}} & (1)\end{matrix}$

Vectors H=(h₁₁, h₁₂, h₁₃, h₂₁, h₂₂, h₂₃, h₃₁, h₃₂, h₃₃) are parametersdepending upon the locations and positions of the individual cameraswith respect to the ground plane and upon the focal lengths and imageorigins of the lenses of the individual cameras. The vectors H arepredetermined from the projected points (u_(i), v_(i)) and (u_(i)′,v_(i)′) (i=1, 2, . . . , and N) of four or more points of the groundplane on the left and right images. By using these relations, thecorresponding point P′(u′, v′) on the right image is determined when itis assumed that an arbitrary point P(u, v) on the left image is presenton the ground plane.

If the point P is present on the ground plane, the points P and P′ makea set of the correct corresponding points so that their intensities areequal. When the points P and P′ have different intensities, therefore,it can be decided that the point P belongs to the obstacle region.According to this method, whether an arbitrary point on the image arisesfrom the ground plane can be decided directly only from Equation (1),and the coefficient of Equation (1) can be determined only from theprojected points of four or more featuring points of the ground upon theleft and right images, thereby to make unnecessary the correspondingpoint search between the left and right images and the cameracalibration using the sample point having a known three-dimensionallocation.

In the case of traveling at a relatively low speed on the flat floorplane in the indoor circumstance, the vectors h can be deemed as fixed,so that the obstacle can be correctly detected by using theonce-determined vectors h.

When the vehicle travels outdoors, however, the relations between theground plane and the relative locations and positions of the individualcameras are changed time after time by the vibrations of the vehicleitself and by the inclination of the ground.

As a result, the parameter vectors h also change with the movement ofthe vehicle. If the obstacle is detected either by using the vectors hdetermined at the still time as they are or with the mere differencefrom the other camera image, therefore, a pattern on the ground iserroneously detected as the obstacle because of the influences of theerrors contained in the parameters, thus causing a problem that thedetection accuracy is seriously lowered.

Here will be described one example of the actions of the differencedetection unit with reference to FIGS. 12 to 16.

First of all, a vehicle carrying two cameras 10 a and 10 b having lightreceiving units 10 a 1 and 10 b 1 spaced at a predetermined distance arearranged on a flat ground plane having no inclination, as shown in FIG.12 presenting a perspective view for explaining the actions. On theground plane, two white lines are drawn, as designated by 1 and 1′, inparallel with each other in the running direction of the vehicle.

It is assumed that the mutual locations and positional relations ofthose two cameras 10 a and 10 b are unknown to the obstacle detectionsystem where as only the epipolar constraint is known, and that thelocations, positions and epipolar constraint of the cameras 10 a and 10b are invariable during the traveling of the vehicle.

Here, the epipolar constrain holds for the general stereo images andspecifies the state in which an arbitrary point P of the (left) imagetaken by the camera 10 a is constrained to exist on a straight linecontaining a corresponding point P′ of the (right) image taken by thecamera 10 b, as illustrated in FIG. 13 for explaining the epipolarconstraint. This straight line is called the “epipolar line”.

When the individual cameras are arranged to have their optical axes inparallel, for example, the corresponding point of the arbitrary point Pof the left image exists on the same scan-line in the right image sothat the epipolar line and the scan-line are aligned with each other.The epipolar constraint depends upon the relative locations andpositional relations between the stereo cameras and upon the intrinsicparameters of the individual cameras, i.e., the focal lengths and theprincipal points of the camera lenses. Therefore, no change in theepipolar constraint means that the relative locational and positionalrelations and the intrinsic parameters of the stereo cameras do notchange during the traveling of the vehicle.

This epipolar constraint is formulated into the following Equation (2):

(u,v,1)F ₍ u,v,1)^(T)=0  (2)

wherein the arbitrary point P of the left image is expressed by (u, v)and the corresponding point of the right image is expressed by (u′, v′).

The letter F indicates a matrix of 3×3, as called the “fundamentalmatrix”. The Equation (2) is expressed by the following Equation (3) bydeveloping and rearranging it:

(F ₁₁ u+F ₁₂ v+F ₁₃)u′+(F ₂₁ u+F ₂₂ +F ₂₃)v′+(F ₃₁ u+F ₃₂ v+F ₃₃)=0  (3)

Equation (3) expresses the epipolar line on the right image, ascorresponding to the point (u, v) of the left image. Here, F_(ji) (i,j=1, 2 and 3) designates an element of j-th row and i-th column of thematrix F and is predetermined from a set of corresponding points.

The matrix F is composed of nine elements, which are not independent butcan be theoretically determined from seven or more pointcorrespondences. The three-dimensional locations of the set of theindividual corresponding points are unnecessary so that the matrix F,i.e., the epipolar constraint can be computed relatively easily.

The lines l and l′ in the individual images are parallel to each otherin the three-dimensional space but intersect in the infinity called the“vanishing point” in the screens, as indicated in the white line regionstaken by the individual cameras in FIG. 14.

Here will be determined a relation to hold between the correspondingpoints of the ground plane. As shown in FIG. 15 for explaining thecorresponding points, two arbitrary points on the straight line l in theleft image are designated by A and C, and two arbitrary points on thestraight line l′ are designated by B and D.

The corresponding points A′, B′, C′ and D′ of those four points on theright image can be easily computed by using the epipolar constraintsdetermined in advance. Specifically, the corresponding point A′ of thepoint A is located at the intersection between the straight line l andthe epipolar line LA of the point A in the right image. Likewise, thepoints B′, C′ and D′ can be determined as the intersections of theindividual points B, C and D with individual epipolar lines LB, LC andLD.

The points A, B, C and D and their corresponding points A′, B′, C′ andD′ are given the coordinates (u₁, v₁), (u₂, v₂), (u₃, v₃) and (u₄, v₄),and (u₁′, v₁′), (u₂′, v₂′), (u₃′, v₃′) and (u₄′, v₄′). The followingrelation holds between (u_(i), v_(i)) and (u_(i)′, v_(i)′) (i=1, 2, 3and 4):

$\begin{matrix}{{u_{i}^{\prime} = \frac{{h_{11}u_{i}} + {h_{12}v_{i}} + h_{13}}{{h_{31}u_{i}} + {h_{32}v_{i}} + h_{33}}},{v_{i}^{\prime} = \frac{{h_{21}u_{i}} + {h_{22}v_{i}} + h_{23}}{{h_{31}u_{i}} + {h_{32}v_{i}} + h_{33}}}} & (4)\end{matrix}$

These eight equations are solved for the vector h=(h₁₁, h₁₂, h₁₃, h₂₁,h₂₂, h₂₃, h₃₁, h₃₂ and h₃₃). If one arbitrary solution vector hsatisfies the foregoing Equation (4), the k-times of the vector h alsosatisfies the Equation so that no generality is lost even for h₃₃=1.From the eight equations, therefore, there can be determined the vectorh which is composed of the nine elements.

By using the vector h=(h₁₁, h₁₂, h₁₃, h₂₁, h₂₂, h₂₃, h₃₁, h₃₂ and h₃₃)thus determined, the corresponding point P′(u′, v′) on the right imageof the case, in which it is assumed that the arbitrary point P(u, v) ofthe left image exists on the ground plane, can be determined andexpressed by the following Equation (5):

$\begin{matrix}{{u^{\prime} = \frac{{h_{11}u} + {h_{12}v} + h_{13}}{{h_{31}u} + {h_{32}v} + h_{33}}},{v^{\prime} = \frac{{h_{21}u} + {h_{22}v} + h_{23}}{{h_{31}u} + {h_{32}v} + h_{33}}}} & (5)\end{matrix}$

In the transformation thus made, as in the transformed image example ofFIG. 16, the left camera image (a) of the stereo images is transformedinto the image shown at (c) when taken in the view point of the rightcamera. Of the pixels on the ground plane, specifically, the contactpoints between the tires of the vehicle and the ground plane of FIG. 16are correctly transformed into the corresponding points, but the objecthaving a spatial height is transformed with a falling distortion in theimage.

When the point P(u, v) and the point P′(u′, v′) have intensities IL(u,v) and IR(u′, v′), they make a correct set of corresponding pints if thepoint P(u, v) actually exists on the ground plane, so that theintensities of the points P and P′ basically have the same intensity. Ifthe points P and P′ have different intensities, on the contrary, they donot exist on the ground plane.

With a constant relation between the ground plane and the cameras, thefollowing Formula is made:

D=|I _(L)(u,v)−I _(R)(u′,v′)|  (6)

(wherein | | designate an absolute value). Considering D≠0 or the errorsdue to the difference between the characteristics of the left and rightcameras, the threshold value Thr can be set to decide that the point forD>Thr belongs to the obstacle region.

As a matter of fact, however, various changes such as the vibrations ofthe cameras and the inclination of the ground plane are caused as thevehicle moves, thereby to make it difficult to discriminate the obstaclefrom the foregoing Equation (6), as will be reasoned in the following.Because of a large intensity difference between a land mark (such as the“stop” mark or the “speed limit” mark or the white lines) and the groundplane, the Equation (6) is caused to take a large value in the vicinityof the land mark (=the edge peripheries) even in the absence of theobstacle by the displacement between the geometric relation between theassumed ground plane and the camera (i.e., the relation between thecamera for determining the aforementioned image transformationparameters and the ground plane) and the geometric relation between theactual ground plane and the camera.

In Embodiment 2, too, it is therefore an object to provide an obstacledetection system capable of stably detecting an obstacle existing on theground plane.

Embodiment 2-1

Here will be described the construction of Embodiment 2-1 of theinvention with reference to the accompanying drawings.

FIGS. 17 to 20 are diagrams showing the obstacle detection system ofthis Embodiment.

FIG. 17 is a block diagram showing the Embodiment of the obstacledetection system. This obstacle detection system is mounted on thedriver's own vehicle to be driven by the driver and is constructed toinclude: image input units 1 and 1 (image pickup units) composed of twostereo cameras having light receiving units 10 a 1 and 10 b 1 arrangedat a predetermined spacing; an image storage unit 2 for storing theimages taken by the image input units 1 and 1; a feature extraction unit3 for extracting the white line portions from the plurality of imagesstored in the image storage unit 2; a difference detection unit 4 fordetecting whether or not it is the obstacle region; and a heightcomputation unit 5 for computing the height of the obstacle from thedetection result of the difference detection unit 4 to detect the trueobstacle region.

Here, the detection of an obstacle assumes a situation that apedestrian, a preceding vehicle or an obstacle to exist on the groundplane is to be detected under the conditions of the vibration to occurwhen the own vehicle travels and the change in the inclination on theground plane. It is also assumed that the image taken by the image inputunit 1 arranged on the left hand side is a first image or a left cameraimage where as the image taken by the image input unit 1 arranged on therighthand side is a second image or a right camera image.

Here will be described the actions of the obstacle detection system thusconstructed.

First of all, the region in the traveling direction of the own vehicleis taken simultaneously into to images by using two TV cameras.

Next, the image storage unit 2 stores the two images, as inputted fromthe image input units 1 and 1, in the image memory.

Next, the feature extraction unit 3 extracts such objects, e.g., twowhite lines l and l′ in the first image and the second image stored inthe image storage unit 2 as are disposed on the ground generally inparallel with each other in the region in the traveling direction of theown vehicle, by the edge extracting processing and the Houghtransformation. On the other hand, the extracted straight lines l and l′determine the intersection in the screen as the vanishing point. At thisvanishing point, the straight lines l and l′ intersect in the firstimage and in the second image.

Next, the difference detection unit 4 reads the left camera image or thefirst image stored in the image storage unit 2, to determine thetransformation parameter by the aforementioned method from the epipolarconstraint and the two lines on the ground plane thereby to determined acorresponding region (of one pixel) in the other right camera image withrespect to the arbitrary region (of one pixel) in the left camera imagetaken by one image input unit 1 (e.g., the left stereo camera). At thistime, it is assumed that all the points in the left camera image arepresent on the ground plane. Specifically, the corresponding relation isdetermined by using the epipolar constraint holding in the stereo imagesand the two lines extracted at the individual images by the featureextraction unit 3.

Then, this arbitrary region and its corresponding region are compared intheir intensities. If the comparison results reveal that the regionshave different intensities and that the intensity difference is no lessthan a standard value set by the user, it is decided that the arbitraryregion is the obstacle region, and the obstacle region image is obtainedfrom the result of the computed intensity difference. Here, thisobstacle region image is formed by detecting the region (or pixel)having a height from the ground plane.

Next, the height computation unit 5 estimates the height of each region(or pixel) from the ground plane, from the obstacle region imagedetected by the difference detection unit 4. It is decided by thisestimated height whether or not the obstacle region is a true obstacle.

More specific description will be made with reference to FIG. 18 forexplaining the actions of the height computation unit 5.

Let it be assumed that there is in the obstacle region image an obstacleregion having an intensity difference larger than the standard value setby the user. Here: a point in the lower side of the obstacle region isdesignated by Ub; a point in the upper side is designated by Ut; and thedistance (as taken in the longitudinal direction of the Drawing) betweenthe point Ub and the point Ut is designated by dv. Moreover, thevanishing point at which the straight lines l and l′ intersect isdesignated by U∞, and the distance (as taken in the longitudinaldirection of the Drawing) from the point Ub to the vanishing line (i.e.,the projected image of the ground plane in the infinite distance) isdesignated by V. Here, the obstacle region extracted is composed of aplurality of pixels by using one of square pixels dividing the obstacleregion image longitudinally and transversely, as the minimum unit.

Here, when the image input unit 1 has a small angle of roll, thevanishing line of the ground plane is substantially aligned with thescan-line which passes through the vanishing point U∞ of the twostraight lines l and l′. By the vibration of the image input unit 1during the traveling of the own vehicle and the change in theinclination of the ground, strictly speaking, the vanishing line of theground plane moves up and down in the image. However, the rolling of theimage input unit 1 is far lower than the pitching, and the transverseinclination of the ground is far smaller than the longitudinalinclination, so that the vanishing line of the ground plane can beapproximated by the scanning line passing through the vanishing point u∞of the two straight lines l and l′ even during the traveling of the ownvehicle.

As shown in FIG. 19 for explaining the actions of the differencedetection unit 4, on the other hand, the following approximate Equationholds, if the distance (i.e., the height of the image input unit 1 fromthe ground) in the real space from the ground to the place where theimage input unit 1 is disposed is designated by H and if the height ofthe obstacle (i.e., the height of the obstacle from the ground) isdesignated by h:

h/H≅dv/V  (7).

Here, the lefthand side of Equation (7) indicates the ratio of theheight of the obstacle from the ground to the height of the image inputunit 1 from the ground, that is, the relative height. If this ratio isdesignated by γ, it is expressed as follows:

γ=dv/V  (8).

Therefore, the relative height γ can be determined from the longitudinalsize dv of the obstacle and the vertical distance V between the lowerside point Ub and the vanishing line of the ground plane.

When the height of each obstacle detected on the image from the groundplane is computed by using Equation (8) and is smaller than a presetheight, i.e., a threshold value γmin, this polygonal region detected isnot the true obstacle region but is eliminated as a noise.

Here, the threshold value γmin takes a value to be set by the positionwhere the image input unit 1 is disposed and from the minimum of theheight of the obstacle to be detected, and is set as follows when theheight H of the image input unit 1 from the ground is 1.5 m and when anobstacle having a height of h=1 m or more is to be detected:

γ=h/H=1[m]/1.5[m]≠0.67  (9).

Next, the output unit 6 presents the user the information such as theposition of the true obstacle detected, in terms of a voice, a light (orimage) or a vibration, and transmits the information to the controlsystem for an autonomous traveling.

According to Embodiment 2-1 thus far described, it is possible tocorrectly detect obstacles on the ground plane in spite of vibrations,road inclination, shadows, road textures, and illumination change.

Embodiment 2-2

Here will be described a construction of Embodiment 2-2 of the obstacledetection system of the invention with reference to FIGS. 20 to 23.

In the following individual Embodiments, the same components as those ofEmbodiment 2-1 are designated by the same reference numerals so thattheir repeated description will be omitted.

This Embodiment is characterized: in that the difference detection unitis constructed to include an image transformation unit 4-1 and adifference computation unit 4-2; and in that the left camera image isconverted into the image in the view point of the right camera assumingthat all image points arise from the ground plane and the obstacle areasare detected comparing the right image to the transferred one.

FIG. 20 is a block diagram of Embodiment 2-2. The difference detectionunit 4 is constructed to include: the image transformation unit 4-1 towhich the signal is inputted from the feature detection unit 3; and thedifference computation unit 4-2 for outputting the signal to the heightcomputation unit 5. Here, the image input unit 1, the image storage unit2, the feature extraction unit 3, the height computation unit 5 and theoutput unit 6 are similar in their constructions and actions to those ofEmbodiment 2-1.

The actions of Embodiment 2-2 thus constructed will be described withreference to FIGS. 21 to 23.

In the following, the description will be made on the case in which theimage is transformed from the right camera image, but the transformedimage may be likewise obtained from the left camera image.

An arbitrary point (u, v) in the right image is used as a variable, anda function having an intensity defined for each point is expressed byf(u, v).

As shown in FIG. 20 for explaining the actions of Embodiment 2-2, theleft image and the right image are inputted by the individual imageinput units 1.

The right image is expressed by g(u, v), and its transformed image isexpressed by g′(u, v). Here can be expressed the following Equation:

g′(u,v)=g(u′,v′)  (10).

Here, the term (u′, v′) is determined from Equation (5).

The term g′(u, v) is the left camera image of the case in which anarbitrary point on the right image g(u, v) exists on the ground plane.

As shown in FIG. 22 for explaining the actions of Embodiment 2-2, forexample, if the right camera image is transformed by using theaforementioned Equation, it is possible to obtain the transformed image,as shown on the righthand side of FIG. 22.

As shown in FIG. 23 for explaining the actions of Embodiment 2-2, on theother hand, the projected point of a point existing on the ground planeis identical in the transformed image to the left camera image. But, apoint non-existing on the ground plane, that is, a point of an obstacle(e.g., a preceding vehicle in this case) is projected on a differentposition according to the height from the ground.

By taking a difference for each corresponding pixel value between theleft camera image and the transformed image, therefore, the obstacle onthe ground plane is detected.

If the left camera image is expressed by f(u, v), the followingexpression can be made:

D′=|f(u,v)−g′(u,v)|  (11).

(wherein | | designate an absolute value).

Considering D′≠0 or the errors, it is decided that the point (u, v) forD′>Thr (wherein Thr designates a preset threshold value) belongs to theobstacle region.

On the other hand, the difference detection unit 4 of Embodiment 2-1 hasdetected the difference between the two images by taking the differencein the pixel value between the corresponding pixels between theindividual images, but the difference computation unit 4-2 may detectthe difference between the individual images by setting a window of(2w+1)×(2w+1) (wherein w designates a natural number) for each point andby computing a normalized cross correlation C of the intensities in thewindow.

The correlation C of the point (u, v) of two images F(u, v) and G(u, v)is expressed by the following Formula:

$\begin{matrix}{C = {\frac{1}{N}{\sum\limits_{\eta = {- w}}^{w}{\sum\limits_{\xi = {- w}}^{w}\frac{\left( {{F\left( {{u + \xi},{v + \eta}} \right)} - a_{1}} \right)\left( {{G\left( {{u + \xi},{v + \eta}} \right)} - a_{2}} \right)}{\sigma_{1}\sigma_{2}}}}}} & (12)\end{matrix}$

wherein: N=(2w+1)×(2w+1); a1 and a2 designate the averages of theintensities in the window of the two images; and σ12 and σ22 designatethe variances in the intensities in the window of the two images. Inthis case, it is decided that the point (u, v) for C<Thr belongs to theobstacle region.

The detected obstacle region is sent as the obstacle region image to theheight computation unit 5.

In the Embodiment 2-2 thus far described, the difference in thecharacteristics between the two cameras can be absorbed to detect thetrue obstacle on the ground plane stably.

Embodiment 2-3

Here will be described Embodiment 2-3 of the obstacle detection systemof the invention.

This Embodiment is characterized by making the feature extraction unitunnecessary.

The two TV cameras are fixed in a space other than the own vehicle.Since the TV cameras are not fixed on the moving body such as the ownvehicle, the geometric relations between the TV cameras and the groundplane do not change so that the transformation parameters of the pointon the ground plane between the stereo images and the location of thevanishing line of the ground plane can be made invariable. For example,the user may mount the TV cameras and may once preset the transformationparameters and the vanishing line of the ground plane.

When the geometric relations between the stereo cameras and the groundplane are thus invariable, the feature extraction unit of Embodiments2-1 and 2-2 can be made unnecessary.

<Modification 2-1>

Here, the invention should not be limited to the foregoing Embodimentsbut can naturally be modified in various manners without departing fromthe gist thereof. For example, the image input units are exemplified bythe two TV cameras arranged on the left and right sides. However, theimage input units can be arranged at any locations of the rear portionof the own vehicle, for example, or three or more image input units canbe arranged, if their light receiving units are spaced from each otherand can take the surroundings of the own vehicle simultaneously as theimages.

<Modification 2-2>

The feature extraction unit sets the transformation parameters bydetermining the corresponding relations of the arbitrary four points butcan use five sets or more corresponding relations. In this case, ten ormore simultaneous equations may be solved by using the least squaremethod.

<Modification 2-3>

The region to be extracted by the feature extraction unit has beendescribed assuming the two white straight lines indicating the lane tobe traveled. When the ground is curved, however, the white lines arealso curved. In this case, the obstacle can be detected likewise byextracting the white lines as the curves.

<Modification 2-4>

The ground plane is assumed to be flat, but the obstacle can be detectedlikewise for the flat plane even if the ground plane has a verticalcurve.

<Modification 2-5>

An automobile or a motorbike has been assumed as the driver's ownvehicle. However, the obstacle detection system can be mounted on anobject of an aeroplane or helicopter to detect an obstacle existing on atakeoff or landing place.

<Modification 2-6>

The obstacle detection system has been exemplified by arranging twocameras at a spacing. It is, however, natural that the obstacledetection system is modified to arrange a plurality of light receivingunits optically separately but to concentrate the cameras at one place.

Embodiment 3

An obstacle detection system according to Embodiment 3 of the inventionis constructed to comprise: a first image pickup unit and a second imagepickup unit for obtaining a first image information of a first image anda second image information of a second image, respectively, by takingthe surrounding region of a driver's own vehicle substantiallysimultaneously as images formed of a set of pixels from light receivingunits arranged at a spacing on the own vehicle; an image informationstorage unit for storing the first image information and the secondimage information; an intensity difference image forming unit forforming an intensity difference image by determining the correspondingpixels in the second image of the second image information, as assumingthat an arbitrary pixel of the first image of the first imageinformation stored in the image information storage unit exists on theground plane being traveled by the own vehicle, to determine theintensity difference between the arbitrary pixel and the correspondingpixel; a discrimination image forming unit for obtaining adiscrimination image by discriminating each pixel in the intensitydifference image into a pixel having an intensity difference no lessthan a standard value and a pixel having an intensity difference lessthan the standard value; and a decision unit for detecting and decidinga region having a generally wedge-shaped set of pixels in thediscrimination image as an obstacle region. Here, the image informationindicates an image composed of a plurality of pixels, and electricsignals transformed from the image.

Here will be described the actions of the aforementioned detection unitwith reference to FIGS. 24 to 28.

Here, the numbers of Equations are independent of those of Embodiments 1and 2 and are newly started from (1).

The actions of the pixel corresponding unit are specified bytransforming the first image taken by the first image pickup unit intothe image, as seen from the view point of the second image pickup unit,thereby to obtain the transformed image.

Here, the parameters to be used for this transformation are assumed: tobe once obtained in advance such that a plurality of image pickup unitsor cameras and the ground plane being traveled by the driver's ownvehicle have a typical geometric relation (for example, when a stillvehicle is arranged on the ground plane having no inclination); and tobe unchanged during the traveling of the vehicle, i.e., while theactions to detect the obstacle are being made.

First of all, a vehicle carrying two cameras 10 a and 10 b are arrangedon a flat ground plane having no inclination, as shown in FIG. 24presenting a perspective view for explaining the actions. On the groundplane, two white lines are drawn, as designated by 1 and 1′, in parallelwith each other in the running direction of the vehicle.

It is assumed that the mutual locations and positional relations ofthose two cameras 10 a and 10 b are unknown to the obstacle detectionsystem where as only the epipolar constraint is known, and that thelocations, positions and epipolar constraint of the cameras 10 a and 10b are invariable during the traveling of the vehicle.

Here, the epipolar constraint holds for the general stereo images andspecifies the state in which an arbitrary point P of the (left) imagetaken by the camera 10 a is constrained to exist on a straight linecontaining a corresponding point P′ of the (right) image taken by thecamera 10 b, as illustrated in FIG. 25 for explaining the epipolarconstraint. This straight line is called the “epipolar line”.

When the individual cameras are arranged to have their optical axes inparallel, for example, the corresponding point of the arbitrary point Pof the left image exists on the same scan-line in the right image sothat the epipolar line and the scan-line are aligned with each other.The epipolar constraint depends upon the geometric relationship betweenthe stereo cameras and upon the intrinsic parameters of the individualcameras, i.e., the focal lengths and the principal points. Therefore, nochange in the epipolar constraint means that the relative locational andpositional relations and the intrinsic parameters of the stereo camerasdo not change during the traveling of the vehicle.

This epipolar constraint is formulated into the following Equation (2):

(u,v,1)F(u,v,1)^(T)=0  (2)

wherein the arbitrary point P of the left image is expressed by (u, v)and the corresponding point of the right image is expressed by (u′, v′).

The letter F indicates a matrix of 3×3, as called the “fundamentalmatrix”. The Equation (2) is expressed by the following Equation (3) bydeveloping and rearranging it:

(F ₁₁ u+F ₁₂ v+F_)u′+(F ₂₁ u+F ₂₂ v+F ₂₃)v′+(F ₃₁ u+F ₃₂ v+F ₃₃)=0  (3)

Equation (3) expresses the epipolar line on the right image, ascorresponding to the point (u, v) of the left image. Here, F_(ji) (i,j=1, 2 and 3) designates an element of j-th row and i-th column of thematrix F and is predetermined from a set of a plurality of correspondingpoints.

The matrix F is composed of nine elements, which are not independent butcan be theoretically determined from a set of seven or morecorresponding points. The three-dimensional locations of the set of theindividual corresponding points are unnecessary so that the matrix F,i.e., the epipolar constraint can be computed relatively easily.

The lines l and l′ are parallel to each other in the three-dimensionalspace but intersect in the so-called “vanishing point” on each image, asindicated in the white line regions taken by the individual cameras inFIG. 26.

Here will be determined a relation to hold between the correspondingpoints of the ground plane. As shown in FIG. 27 for explaining thecorresponding points, two arbitrary points on the straight line l in theleft image are designated by A and C, and two arbitrary points on thestraight line l′ are designated by B and D.

The corresponding points A′, B′, C′ and D′ of those four points on theright image can be easily computed by using the epipolar constraintsdetermined in advance. Specifically, the corresponding point A′ of thepoint A is located at the intersection between the straight line l andthe epipolar line LA of the point A in the right image. Likewise, thepoints B′, C′ and D′ can be determined as the intersections of theindividual points B, C and D with individual epipolar lines LB, LC andLD.

The points A, B, C and D and their corresponding points A′, B′, C′ andD′ are given the coordinates (u₁, v₁), (u₂, v₂), (u₃, v₃) and (u₄, v₄),and (u₁′, v₁′), (u₂′, v₂′), (u₃′, v₃′) and (u₄′, v₄′). The followingrelation holds between (u_(i), v_(i)) and (u_(i)′, v_(i)′) (i=1, 2, 3and 4):

$\begin{matrix}{{u_{i}^{\prime} = \frac{{h_{11}u_{i}} + {h_{12}v_{i}} + h_{13}}{{h_{31}u_{i}} + {h_{32}v_{i}} + h_{33}}},{v_{i}^{\prime} = \frac{{h_{21}u_{i}} + {h_{22}v_{i}} + h_{23}}{{h_{31}u_{i}} + {h_{32}v_{i}} + h_{33}}}} & (4)\end{matrix}$

These eight equations are solved for the vector h=(h₁₁, h₁₂, h₁₃, h₂₁,h₂₂, h₂₃, h₃₁, h₃₂ and h₃₃). If one arbitrary solution vector hsatisfies the foregoing Equation (4), the k-times of the vector h alsosatisfies the Equation so that no generality is lost even for h₃₃=1.From the eight equations, therefore, there can be determined the vectorh which is composed of the nine elements.

By using the vector h=(h₁₁, h₁₂, h₁₃, h₂₁, h₂₂, h₂₃, h₃₁, h₃₂ and h₃₃)thus determined, the corresponding point P′ (u′, v′) on the right imageof the case, in which it is assumed that the arbitrary point P(u, v) ofthe left image exists on the ground plane, can be determined andexpressed by the following Equation (5):

$\begin{matrix}{{u^{\prime} = \frac{{h_{11}u} + {h_{12}v} + h_{13}}{{h_{31}u} + {h_{32}v} + h_{33}}},{v^{\prime} = \frac{{h_{21}u} + {h_{22}v} + h_{23}}{{h_{31}u} + {h_{32}v} + h_{33}}}} & (5)\end{matrix}$

In the transformation thus made, as in the transformed image example ofFIG. 28, the left camera image (a) of the stereo images is transformedinto the image shown at (c) when taken in the view point of the rightcamera. Of the pixels on the ground plane, specifically, the contactpoints between the tires of the vehicle and the ground plane of FIG. 16are correctly transformed into the corresponding points, but the objecthaving a spatial height is transformed with a falling distortion in theimage.

When the point P(u, v) and the point P′(u′, v′) have intensitiesI_(L)(u, v) and IR(U′, V′), they make a correct set of correspondingpints if the point P(u, v) actually exists on the ground plane, so thatthe intensities of the points P and P′ basically have the sameintensity. If the points P and P′ have different intensities, on thecontrary, they do not exist on the ground plane.

With a constant relation between the ground plane and the cameras, thefollowing Formula is made:

D=|I _(L)(u,v)−I _(R)(u′,v′)|  (6)

(wherein | | designate an absolute value). Considering D≠0 or the errorsdue to the difference between the characteristics of the left and rightcameras, the threshold value Thr can be set to decide that the point forD>Thr belongs to the obstacle region.

As a matter of fact, however, various changes such as the vibrations ofthe cameras and the inclination of the ground plane are caused as thevehicle moves, thereby to make it difficult to discriminate the obstaclefrom the foregoing Equation (6), as will be reasoned in the following.Because of a large intensity difference between a land mark (such as the“stop” mark or the “speed limit” mark or the white lines) and the groundplane, the Equation (6) is caused to take a large value in the vicinityof the land mark (=the edge peripheries) even in the absence of theobstacle by the difference between the geometric relation between theassumed ground plane and the camera (i.e., the relation between thecamera for determining the aforementioned image transformationparameters and the ground plane) and the geometric relation between theactual ground plane and the camera.

Embodiment 3-1

Here will be described the construction of Embodiment 3-1 of theobstacle detection system of the invention with reference to theaccompanying drawings.

FIGS. 29 to 33 are diagrams showing the Embodiment 3-1 of the obstacledetection system of the invention.

FIG. 29 is a block diagram showing Embodiment 3-1 of the obstacledetection system. This obstacle detection system is mounted on adriver's own vehicle 10 (as should be referred to FIG. 30) to be drivenby the driver and is constructed to include: image input units 1 and 1(image pickup units) composed of two stereo cameras having lightreceiving units or lenses 1 a and 1 b arranged at a spacing; an imageinformation storage unit 2 for storing the image informations taken bythe image input units 1 and 1; a detection unit 3 for detecting theobstacle from the taken images by processing (as will be detailed) thetaken images, to obtain the detected result as the obstacle regionimage; and a vehicle detection unit having a detection unit fordetecting a generally wedge-shaped region from the obstacle regionimages.

Here, the detection of an obstacle assumes a situation that apedestrian, a preceding vehicle or an obstacle to exist on the groundplane is to be detected under the conditions of the vibration to occurwhen the own vehicle travels and the change in the inclination on theground plane. It is also assumed that the geometrical relation of thetwo cameras and the construction at the time when the stereo cameras aremounted on the own vehicle are neither changed nor varied from those ofthe aforementioned time when the image transformation parameters arecomputed. It is further assumed that the image taken by the image inputunit 1 arranged on the lefthand side is a first image or a left cameraimage where as the image taken by the image input unit 1 arranged on therighthand side is a second image or a right camera image.

The locations for mounting the image input units 1 and 1 should not belimited to the ceiling side of the vehicle, but the image input units 1and 1 may be fixed at any located in the vehicle, as exemplified by theobstacle detection system of Embodiment 3-1 of FIG. 30 if the drive ofthe user is not troubled and if the running direction (as indicated byan arrow 11) of the own vehicle 10. However, the locations for fixingthe image input units 1 and 1 should not change during the traveling ofthe own vehicle.

Here will be described the actions of the obstacle detection system thusconstructed.

First of all, the region in the traveling direction of the own vehicleis taken simultaneously into to images by using two TV cameras.

Next, the image information storage unit 2 stores the two images, asinputted from the image input units 1 and 1, in the image memory.

Next, in the detection unit 3, as shown in FIG. 31 for explaining theactions of the detection unit of Embodiment 3-1, the pixel correspondingunit 3 a reads the left camera image or the first image (or the imageinformation), as stored in the image information storage unit 2, anddetermines the corresponding pixel in the right camera imagecorresponding to a predetermined pixel in the left camera image, on thebasis of the relation (i.e., the ground plane constraint) holdingbetween the projected locations of an arbitrary pixel on the groundplane predetermined at the still time of the own vehicle, thereby toperform the matching processing (as should be referred to FIG. 31(C)).At this time, assuming that the arbitrary pixel exists on the groundplane in the left camera image, the corresponding pixel in the rightcamera image is determined. This matching processing is performed forall the pixels that individually construct the left camera image and theright camera image.

The intensity difference image forming unit 3 b determines the intensitydifference of the corresponding pixel, as corresponding to the pixel inthe left camera image, of the right camera image.

The discrimination image forming unit 3 c discriminates whether theintensity difference of each pixel in the intensity difference image isno less or less than the standard value preset by the user. As a resultof this discrimination, the pixel having an intensity no less than thestandard value belongs to the obstacle region, and the pixel having anintensity no more than the standard value belongs not to the obstacleregion but to the ground plane, thereby to divide the obstacle regionand the non-obstacle region such as the ground plane. Here, theintensity difference of each corresponding pixel, as obtained from theleft camera image and the right camera image, is obtained as thediscrimination image.

Next, the vehicle detection unit 5 detects the region, in which the setof pixels has a general wedge shape, from the discrimination image, anddecides the detected region as the obstacle region. The reason why theregion for detecting obstacle is formed into the general wedge shape isempirical but is that when the left camera image is transformed into theright camera viewpoint, the intensity of the pixel in the region havingan increasing height on the side of the vehicle is not equalized butformed into the general wedge shape. On the other hand, this wedge shapeis intensified on the two ends of the obstacle in the transversedirection of the image.

The region of the general wedge shape is detected from thediscrimination image by scanning the wedge-shaped template stored in thevehicle detection unit 5, as shown in FIG. 33, in the raster directionof the discrimination image to perform the template matching. In thistemplate matching, the size of the temperature used is smaller than theobstacle region to be detected.

Here, the wedge-shaped region and the template are so arranged that thelocational relation between a side H existing generally in the scan-linedirection of the left camera image (or the right camera image) and anapex P opposed to the side H has the side H at an upper location of theimage.

The lower location of the image of the general wedge shape in thediscrimination image, as detected by the vehicle detection unit 5, thatis, the location of the apex P is at the contact between the obstacleregion in the traveling direction of the own vehicle and the groundbeing traveled by the own vehicle, or at a portion of the obstacleregion the closest to the own vehicle.

On the other hand, the generally wedge-shaped region detected by thevehicle detection unit 5 exists in one pair in the generally identicalshape at a spacing on the generally common scan-line so that the regiondefined by these paired generally wedge-shaped regions is decided to bethe obstacle region.

On the other hand, the obstacle information, as detected by the vehicledetection unit 5, can be suitably provided for the user by the audiomeans such as voices, the visual means such as lights (including theimages), or the bodily sensation means such as vibrations.

According to Embodiment 3-1 thus far described, no matter what groundthe own vehicle might travel, the obstacle can be stably detectedwithout being influenced by the fluctuation of the brightness or theshadow of the preceding vehicle while suppressing the influences of thevibration and the inclination of the ground itself. By warning thepresence of the obstacle to the user, on the other hand, it is possibleto avoid the event which may occur in the presence of the obstacle, asthe user desires so.

Embodiment 3-2

Here will be described a construction of Embodiment 3-2 of the obstacledetection system of the invention with reference to FIGS. 34 to 37.

In the following Embodiment 3-2, the same components as those ofEmbodiment 3-1 are designated by the same reference numerals so thattheir repeated description will be omitted.

This Embodiment 3-1 is characterized: in that the detection unit isconstructed to include an image transformation unit 3-1 and a differencecomputation unit 3-2; and in that the left camera image is convertedinto the image in the view point of the right camera so that theobstacle is detected by comparing an arbitrary region of the transformedimage and the corresponding region, as corresponding to the arbitraryregion, of the right camera image.

FIG. 34 is a block diagram of Embodiment 3-2. The detection unit 3 isconstructed to include: the image transformation unit 3-1 to which thesignal is inputted from the image information storage unit 2; and thedifference computation unit 3-2 for outputting the signal to the vehicledetection unit 4. Here, the image input unit 1, the image informationstorage unit 2 and the vehicle detection unit 4 are similar in theirconstructions and actions to those of Embodiment 3-1.

The actions of Embodiment 3-2 thus constructed will be described withreference to FIGS. 35 to 37.

In the following, the description will be made on the case in which theimage is transformed from the right camera image, but the transformedimage may be likewise obtained from the left camera image.

An arbitrary point (u, v) in the right image is used as a variable, anda function having an intensity defined for each point is expressed byf(u, v).

As shown in FIG. 34 for explaining the actions of Embodiment 3-2, theleft image and the right image are inputted by the individual imageinput units 1.

The right image is expressed by g(u, v), and its transformed image isexpressed by g′(u, v). The following Equation is determined fromEquation (5):

g(u,v)=g(u′,v′)  (10).

The term g′ (u, v) is the left camera image of the case in which anarbitrary point on the right image g(u, v) exists on the ground plane.

As shown in FIG. 36 for explaining the actions of Embodiment 3-2, forexample, if the right camera image is transformed by using theaforementioned Equation, it is possible to obtain the transformed image,as shown on the righthand side of FIG. 36.

As shown in FIG. 37 for explaining the actions of Embodiment 3-2, on theother hand, the projected point of a point existing on the ground planeis identical in the transformed image to the left camera image. But, apoint non-existing on the ground plane, that is, a point of an obstacle(e.g., a preceding vehicle in this case) is projected on a differentposition according to the height from the ground.

By taking a difference for each corresponding pixel value between theleft camera image and the transformed image, therefore, the obstacle onthe ground plane is detected. If the left camera image is expressed byf(u, v), the following expression can be made:

D′=|f(u,v)−g′(u,v)|  (11).

(wherein | | designate an absolute value).

Considering D′≠0 or the errors, it is decided that the point (u, v) forD′>Thr (wherein Thr designates a preset threshold value) belongs to theobstacle region.

On the other hand, the detection unit 4 of Embodiment 3-1 has detectedthe difference between the two images by taking the difference in thepixel value between the corresponding pixels between the individualimages, but the difference computation unit 3-2 may detect thedifference between the individual images by setting a window of(2w+1)×(2w+1) (wherein w designates a natural number) for each point andby computing a normalized cross correlation C of the intensities in thewindow.

The correlation C of the point (u, v) of two images F(u, v) and G(u, v)is expressed by the following Formula:

$\begin{matrix}{C = {\frac{1}{N}{\sum\limits_{\eta = {- w}}^{w}{\sum\limits_{\xi = {- w}}^{w}\frac{\left( {{F\left( {{u + \xi},{v + \eta}} \right)} - a_{1}} \right)\left( {{G\left( {{u + \xi},{v + \eta}} \right)} - a_{2}} \right)}{\sigma_{1}\sigma_{2}}}}}} & (12)\end{matrix}$

wherein: N=(2w+1)×(2w+1); a₁ and a₂ designate the averages of theintensities in the window of the two images; and σ₁ ² and σ₂ ² designatethe variances in the intensities in the window of the two images. Inthis case, it is decided that the point (u, v) for C<Thr belongs to theobstacle region.

The detected obstacle region is sent as the obstacle region image to thevehicle detection unit 5.

According to Embodiment 2-1 thus far described, it is possible tocorrectly detect obstacles on the ground plane in spite of vibrations,road inclination, shadows, road textures, and illumination change.

<Modification 3-1>

Here, the invention should not be limited to the foregoing Embodimentsbut can naturally be modified in various manners without departing fromthe gist thereof.

For example, the image input units are exemplified by the two TVcameras, but the image input units can be arranged at any locations, orthree or more image input units can be arranged, if their lightreceiving units are arranged to have parallel optical axes and spacedfrom each other and can take the forward regions of the own vehiclesimultaneously as the images.

<Modification 3-2>

The feature extraction unit sets the transformation parameters bydetermining the corresponding relations of the arbitrary four sets ofpoints but can use five sets or more corresponding relations. In thiscase, ten or more simultaneous equations may be solved by using themethod of least squares.

<Modification 3-3>

The ground plane is assumed to be flat, but the obstacle can be detectedlikewise for the flat plane even if the ground plane has a verticalcurve with respect to the ground surface.

<Modification 3-4>

An automobile or a motorbike has been assumed as the driver's ownvehicle. However, the obstacle detection system can be mounted on anobject of an aeroplane or helicopter to detect an obstacle existing on atakeoff or landing place.

<Modification 3-5>

The obstacle detection system has been exemplified by arranging twocameras at a spacing. It is, however, natural that the obstacledetection system is modified to arrange a plurality of light receivingunits optically separately but to concentrate the cameras at one place.

<Modification 3-6>

The ground plane has been described assuming it to be fat. Even if theground plane is assumed to be curved, however, the obstacle can bedetected, for example, by approximating the curve into a plurality ofsectional planes, by preparing a plurality of image transformationparameters and by performing the image transformation sectionally on theimage.

Embodiment 4

Here will be described Embodiment 4 of the invention with reference toFIGS. 39 to 49.

Here, the numbers of Equations are independent of those of Embodiments1, 2 and 3 and are newly started from (1).

In Embodiment 4, there are imagined situations that an obstacle existingon a ground plane such as a pedestrian, a preceding vehicle or a parkedvehicle is detected by two left and right stereo cameras mounted on avehicle (as will be called the “own vehicle”), as shown in FIG. 39, andthat the time for the obstacle to contact with the own vehicle iscomputed.

FIG. 40 shows a schematic construction of the obstacle detection systemaccording to Embodiment 4. This obstacle detection system is constructedto include an image input unit 1, an image storage unit 2, a featureextraction unit 3, a detection unit 4 and a contact time computationunit 5.

In the obstacle detection system, a stereo images are obtained by twocameras, the mutual locations/positions and the focal lengths/principalpoints of the lenses are unknown, and an equation (as will be called the“ground plane constraint”) expressing a relation to hold between theprojected positions of a point of the ground plane, as determined at astill time, upon the projected locations is used to discriminate whetheror not each point on the image has a height from the ground plane,thereby to separate the obstacle region and the ground region. Moreover,the time for the obstacle to contact with the own vehicle is computedfrom the locus of the obstacle on the image.

FIG. 41 shows a coordinate system of the own vehicle.

In the own vehicle coordinate system: the advancing (or longitudinal)direction of the own vehicle is taken in the Y-axis; the transverse andvertical directions are taken in the X-axis and the Z-axis,respectively; and the ground plane is taken in the X-Y plane. InEmbodiment 4, it is premised that both the own vehicle and the obstacletravel along the two white lines (or straight lines l and l′) on the tworoad ends.

(Image Input Unit 1)

The image input unit 1 takes two images by using two left and right TVcameras. The locations and positions of those two cameras with respectto the own vehicle coordinate system and the focal lengths and theprincipal points of the lenses of the individual cameras may be unknown,but it is assumed in Embodiment 4 that the individual cameras are fixedon the vehicle and not changed during the traveling.

(Image Storage Unit 2)

The image storage unit 2 stores an image memory with the two imagesinputted from the image input unit 1.

(Feature Extraction Unit 3)

The feature extraction unit 3 detects the two straight lines 11 and 12,as shown in FIG. 42, by using one of the two images stored in the imagestorage unit 2, as a standard image (which will be exemplified by theleft image in the following).

The point of intersection (or the projected point of the infinite pointsof the two straight lines upon the images, as will be called the“vanishing point”) is expressed by u₀(u₀, v₀). This straight linedetection is performed by using the edge extraction processing and theHough transformation.

(Detection Unit 4)

The detection unit 4 detects the obstacle by using the constraint (aswill be called the “ground plane constraint”) to hold between theprojected points of the point of the ground plane upon the left andright images. Here will be described this ground plane constraint.

If the projected points of an arbitrary point (X, Y) of the ground plane(or the X-Y plane) upon the left and right images are designated by (u,v) and (ur, vr), the following relation generally holds:

$\begin{matrix}{{u = \frac{{h_{11}X} + {h_{12}Y} + t_{1}}{{h_{31}X} + {h_{32}Y} + t_{3}}},\mspace{25mu} {\upsilon = \frac{{h_{21}X} + {h_{22}Y} + t_{2}}{{h_{31}X} + {h_{32}Y} + t_{3}}}} & (1) \\{{u_{r} = \frac{{h_{11}^{\prime}X} + {h_{12}^{\prime}Y} + t_{1}^{\prime}}{{h_{31}^{\prime}X} + {h_{32}^{\prime}Y} + t_{3}^{\prime}}},\mspace{20mu} {\upsilon_{r} = \frac{{h_{21}^{\prime}X} + {h_{22}^{\prime}Y} + t_{1}^{\prime}}{{h_{31}^{\prime}X} + {h_{32}^{\prime}Y} + t_{3}^{\prime}}}} & (2)\end{matrix}$

wherein h=(h₁₁, h₁₂, . . . , and t₃)^(T) and h′=(h′₁₁, h′₁₂, . . . , andt′₃)^(T) are parameters relating to the three-dimensional locations andpositions of the individual cameras with respect to the own vehiclecoordinate system, and the focal lengths and the principal points of thelenses mounted on the individual cameras.

If X and Y are eliminated from Equations (1) and (2), the followingFormula is obtained:

$\begin{matrix}{{u_{r} = \frac{{H_{11}u} + {H_{12}\upsilon} + H_{13}}{{H_{31}u} + {H_{32}\upsilon} + H_{33}}},\mspace{20mu} {\upsilon_{r} = \frac{{H_{21}u} + {H_{22}\upsilon} + H_{13}}{{H_{31}u} + {H_{32}\upsilon} + H_{33}}}} & (3)\end{matrix}$

wherein H=(H₁₁, H₁₂, . . . , and H₃₃)^(T) are constants expressed by hand h′.

From this Formula, there can be determined the corresponding point onthe right image when the point (u, v) on the left image is assumed to beon the ground plane.

In other words, assuming that the point (u, v) on the left image is onthe ground plane, the corresponding point (u_(r), v_(r)) on the rightimage is determined by the aforementioned Equation. This Equation willbe called the “ground plane constraint”.

Here, the parameters H=(H₁₁, H₁₂, . . . , and H₃₃)^(T) are predeterminedat the still time.

Here will be described this method.

First of all, an N (≧4) number of featuring points (e.g., theintersection point of the straight lines drawn on the ground plane orthe corner points of paint) on the ground plane are extracted from theleft image.

Next, there are determined the corresponding points of the individualextracted featuring points on the right image. Here, these featuringpoint extractions and the correspondences may also be performed bypointing the points on the images with the mouse. These N sets ofcorresponding relations satisfy the Equation (3) individually so that a2N number of simultaneous equations are obtained.

The parameter H can be determined by solving those simultaneousequations for H.

Here will be described the method for detecting the obstacle by usingthe ground plane constraint.

The corresponding point A′ (u_(r), v_(r)) on the right image of the casein which it is assumed that the brightness of an arbitrary point A(u, v)of the left image has an intensity IL (U, v) and that the point A existson the ground plane is determined from Equation (3) to have an intensityI_(R)(u_(r), v_(r)).

If the point (u, v) is actually present on the ground plane, the pointsA and A′ make a set of correct corresponding points (that is, the pointsA and A′ are the projected points of the same point of the ground plane)so that the point A and the point A′ basically have an equal intensity.For the following Formula, therefore, Diff=0:

Diff=|I _(L)(u,v)−I _(R)(u _(r) ,v _(r))|(|·|:Absolute value)  (4)

Unless Diff=0, on the contrary, the point A(u, v) and the point A′(u_(r), v_(r)) do not belong to the set of correct correcting points, sothat the point (u, v) is not present on the ground plane, i.e., thepoint on the obstacle.

After these series of processing were performed on all the points of thestandard image, it is possible to detect the obstacle region.

At this time, considering an error of some extent as the standard fordeciding the obstacle, it may be decided that a point for Diff>Thrbelongs to the obstacle region.

Here, Thr designate a preset threshold value.

From the stereo images shown in FIG. 43, for example, there is detectedan obstacle region, as shown in the lower side of FIG. 43.

(Contract Time Measurement Unit 5)

The contact time measurement unit 5 measures the time (i.e., the time tocontact) for the own vehicle to contact with the obstacle, from thelocus (or the time change of the location of the obstacle) of theobstacle on one image (as will be called the “standard image” which isexemplified by the left image in Embodiment 4) of the stereo images.

First of all, the time to contact will be described in the following.

The denominator of Equation (1) is designated by D. That is:

D=h ₃₁ X+h ₃₂ Y+t ₃  (5).

The denominator D indicates the distance, i.e., the depth in the opticalaxis between a contact point T (X, Y) of the obstacle with the groundplane and the viewpoint C of the standard camera.

The depths of the contact points T′ and T of the obstacle with theground plane at instants t-dt and t are designated by D′ and D,respectively. Here, the time period, for which the depth becomes 0 whenthe obstacle continues movements between the instants t-dt and t withrespect to the own vehicle coordinate system, is defined by a time tocontact t_(c).

FIG. 45 shows the time change in the depth.

The time period from the present time t to the time for the depth tobecome 0 is the time to contact t_(c), as illustrated in FIG. 45. Fromthe similar relation of the two hatched triangles, the followingrelation is obtained:

$\begin{matrix}{{{from}\mspace{14mu} \frac{t_{c}}{D}} = \frac{dt}{D^{\prime} - D}} & (6) \\{\; {t_{c} = {{\frac{D}{D^{\prime} - D}{dt}} = {{\frac{1}{{D^{\prime}/D} - 1}{dt}} = {\frac{1}{\gamma - 1}d}}}}} & (7)\end{matrix}$

wherein γ=D′/D.

In short, the time to contact tc can be determined from the ratio γ ofthe depth between the two times.

Here will be described the method for computing the time to contact,from the positions of the obstacle regions at the individual times, asdetermined by the detection unit 4.

It is assumed that the obstacle region is detected, as shown in FIG. 46,on the standard image at the present time t and the preceding time t-dt,and that an arbitrary point on the lowermost line (or the boundary linebetween the obstacle region and the ground region) detected at the timet is expressed by u=(u, v). The position u′(u′, v′) of the point u=(u,v) at the time t-dt can be determined as the point of intersectionbetween the straight line joining the points u₀ and u and the boundaryline at the time t-dt between the obstacle region and the ground region.

On the other hand, it is assumed that the locations of the obstacle onthe ground plane at the times t-dt and t are expressed by X′=(X, Y+dY)and X=(X, Y).

In Embodiment 4, it is assumed that the own vehicle and the obstaclemove in the Y-axis (or in the same traveling direction), so that theobstacle at individual times is located to have the same X-coordinate.The depth to the point X′ is expressed, as follows:

From formula (1),

$\begin{matrix}{D^{\prime} = {{h_{31}X} + {h_{32}\left( {Y + {dY}} \right)} + t_{3}}} & (8) \\\left\{ \begin{matrix}{{D\; \upsilon} = {{h_{21}X} + {h_{22}Y} + t_{2}}} \\{{D^{\prime}\upsilon^{\prime}} = {{h_{21}X} + {h_{22}\left( {Y + {dY}} \right)} + t_{2}}}\end{matrix} \right. & (9)\end{matrix}$

From the foregoing two formulae as well as formulae (5) and (8),

$\begin{matrix}{{{D^{\prime}\upsilon^{\prime}} - {D\; \upsilon}} = {{h_{22}\left( {Y^{\prime} - Y} \right)} = {\frac{h_{22}}{h_{32}}\left( {D^{\prime} - D} \right)}}} & (10)\end{matrix}$

Here, the projected point of the infinite point on the straight lines land l′ parallel to the Y-axis is expressed by u0=(u0, v0). Y→∞ inEquation (1):

$\begin{matrix}{u = {\left. \frac{{h_{11}{X/Y}} + h_{12} + {t_{1}/Y}}{{h_{31}{X/Y}} + h_{32} + {t_{3}/Y}}\rightarrow\frac{h_{12}}{h_{32}} \right. = u_{0}}} & (11) \\{\upsilon = {\left. \frac{{h_{21}{X/Y}} + h_{22} + {t_{2}/Y}}{{h_{31}{X/Y}} + h_{32} + {t_{3}/Y}}\rightarrow\frac{h_{22}}{h_{32}} \right. = \upsilon_{0}}} & (12)\end{matrix}$

when v₀=h₂₂/h₃₂ is substituted for formula (10),

D′(v′−v ₀)=D(v−v ₀)  (13)

Accordingly

$\begin{matrix}{\gamma = {\frac{D^{\prime}}{D} = \frac{\upsilon - \upsilon_{0}}{\upsilon^{\prime} - \upsilon_{0}}}} & (14)\end{matrix}$

By substituting this value γ into Equation (7), it is possible tocompute the time to contact of the obstacle.

In short, the time to contact of the obstacle to move on the groundplane, as shown in FIG. 46, can be computed exclusively from thelocations u′ and u of the contact point between the obstacle and theground plane at the times t-dt and t and the vanishing point u0 of thestraight lines l and l′ and are not dependent upon the locations of thecameras with respect to the own vehicle coordinate system and thepositions of the cameras.

Thus, neither using the camera parameters nor performing the search ofthe depth, the obstacle on the ground plane can be detected from thestereo cameras on the vehicle, and the time period for the obstacle tocontact with the own vehicle can be computed.

(Effects of Embodiment 4)

According to Embodiment 4, as has been described hereinbefore, thestereo images taken by the stereo cameras are processed to detect theobstacle in dependence upon the presence or absence of the height fromthe ground plane. Without being influenced by the fluctuation of thebrightness or the influence of the shadow, therefore, it is possible todetect the obstacle such as the preceding vehicle or the pedestrian fromthe images. Moreover, it is possible to eliminate the calibrationsrequiring a long time and much work and the depth search (or thecorresponding search) of the high computation cost, which have troubledthe stereo vision of the prior art, so that high practical effects canbe obtained.

<Modification 4-1>

In Embodiment 4, the image input unit 1 inputs two images by arrangingthe two TV cameras transversely, which may be vertically arranged.

On the other hand, there may be arranged three or more cameras.

<Modification 4-2>

The feature extraction unit 3 has been described on the casein which thetwo lines on the ground plane are to be extracted, but three or morelines may be extracted. Alternatively, a line absent from the groundplane, such as the line on a guard rail may be extracted if thedirection vector is identical.

<Modification 4-3>

The detection unit 4 can take a construction shown in FIG. 47.

Here, the construction is made of an image transformation unit 4-1 and adifference computation unit 4-2.

The image transformation unit 4-1 transforms a right image in accordancewith the following procedure. Generally, the image can be expressed as afunction ƒ(u, v) which uses the point (u, v) on the image as a variableand which has an intensity defined for each point. In the following, theimage is thus expressed.

It is assumed that a stereo image shown in FIG. 48 is inputted, and theright image expressed by g(u, v) and the transformed image expressed byg′ (u, v) are determined. This image g′(u, v) is determined in thefollowing manner.

g′(u,v)=g(u _(r) ,v _(r))  (15)

wherein (u_(r), v_(r)) is determined from Equation (3). Here, the imageg′(u, v) is one which is obtained by the left camera when it is assumedthat all the points on the image g(u, v) exist on the ground plane.

From the right image of FIG. 49, for example, there is obtained thetransformed image, as shown in FIG. 49.

As shown in FIG. 50, the projected points of a point on the ground planeare identical between the left image and the transformed image, but apoint non-existing on the ground plane, i.e., a point on the obstacle(i.e., the preceding vehicle in this case) is projected at a locationdifference according to the height from the ground.

Therefore, the obstacle on the ground plane is detected by taking adifference between the left image and the transformed image.Specifically, the left image is expressed by f(u, v). Unless Diff′=0 orif Diff′>Thr (Thr: a preset threshold value) considering the error, thepoint (u, v) belongs to the obstacle region:

Diff′=|f(u,v)−g′(u,v)|(|·|:Absolute value)  (16)

<Modification 4-4>

The detection unit 4 detects the difference between the two images bytaking the pixel difference, but may detect the difference by setting awindow of (2w+1)×(2w+1) for each point, from the average, the dispersionor a normalized cross correlation of the intensities in the window.

The correlation C of the point (u, v) of two images F(u, v) and G(u, v)can be computed from the following Formula:

$\begin{matrix}{C = {\frac{1}{N}{\sum\limits_{\eta = {- w}}^{w}{\sum\limits_{\xi = {- w}}^{w}\frac{\left( {{F\left( {{u + \xi},{\upsilon + \eta}} \right)} - a_{1}} \right)\left( {{G\left( {{u + \xi},{\upsilon + \eta}} \right)} - a_{2}} \right)}{\sigma_{1}\sigma_{2}}}}}} & (17)\end{matrix}$

wherein: N=(2w+1)×(2w+1); a1 and a2 designate the averages of theintensities in the window of the two images; σ₁ ² and σ₂ ² designate thevariances in the intensities in the window of the two images; and−1≦C≦1.

In this case, it is decided that the point (u, v) for C<Thr belongs tothe obstacle region. Here, Thr(≦1) designate a present threshold value.

<Modification 4-5>

In Embodiment 4, the two white lines on the two ground ends areextracted as the straight lines, but the white lines are curves when theground is curved.

In this case, the obstacle can be likewise detected if the white linesare extracted as the curves.

<Modification 4-6>

The description has been made assuming that the ground plane is flat,but the obstacle can be detected as in the case of the flat plane evenfor the curved plane.

<Modification 4-7>

The movements parallel to the lane have been assumed for the own vehicleand the obstacle. However, the obstacle detection system can also beapplied to the case in which the movements are not parallel (e.g., whenthe lane is to be changed).

<Modification 4-8>

Although Embodiment 4 has been described on the obstacle detection fromthe cameras mounted on the vehicle, it could be applied to theautonomous run of a moving robot, for example, but this method shouldnot be limited to the obstacle detection from the vehicle-mountedcameras.

<Modification 4-9>

Although the description has been made on the case in which the stereocameras are disposed in front of the vehicle to detect the forwardobstacle, the cameras can be disposed on the side or at the back of thevehicle to detect the obstacle in each direction.

<Modification 4-10>

The contact time measurement unit 5 can warn the driver in voices of adifferent volume or tone in accordance with the duration of the contacttime when the computed time to contact with the obstacle is short.

<Modification 4-11>

The camera parameters of the vehicle-mounted stereo cameras are assumedto be invariable during the traveling. If variable, however, theobstacle can be detected if the ground plane constraint is updated, tocompute the contact time.

INDUSTRIAL APPLICABILITY

According to the invention, an obstacle is detected depending upon thepresence or absence of the height from the ground plane so that theobstacle such as a preceding vehicle or a pedestrian can be detectedfrom images independently of the fluctuation of brightness or theinfluence of a shadow. On the other hand, the constraint to hold in thegeometric relation between the ground plane and individual cameras isdetermined from only two white lines of the two ground ends so that theobstacle on the ground plane can be quickly detected to provide highpractical effects even with the vibration during the traveling or theinclination of the ground plane.

Even when the vibration occurs in the traveling own vehicle or when theground being traveled has an inclination, on the other hand, the trueobstacle on the ground plane can be stably detected without erroneouslydetecting the pattern drawn on the ground plane being traveled.

Moreover, the stable obstacle detection can be made without beinginfluenced by the vibration to be applied to the image pickup units orthe inclination of the ground plane being traveled.

On the other hand, the images taken by a plurality of cameras areprocessed to detect the obstacle in dependence upon the presence orabsence of the height from a plane in a three-dimensions, so that theobstacle can be detected from the images without being influenced by thefluctuation in the brightness or the shadow. On the other hand, highpractical effects can be obtained because there are unnecessary thecamera calibrations requiring a long time and much work and the depthsearch (or the corresponding search) of the high computation cost, whichhave troubled the stereo vision of the prior art.

1. An obstacle detection system comprising: a first image pickup unitand a second image pickup unit for obtaining a first image informationof a first image and a second image information of a second image,respectively, by taking the surrounding region of a driver's own vehiclesubstantially simultaneously as images formed of a set of pixels fromlight receiving units arranged at a spacing on the own vehicle; an imageinformation storage unit for storing said first image information andsaid second image information; an intensity difference image formingunit for forming an intensity difference image by determining thecorresponding pixels in said second image of the second imageinformation, as assuming that an arbitrary pixel of said first image ofsaid first image information stored in said image information storageunit exists on the ground plane being traveled by said own vehicle, todetermine the intensity difference between said arbitrary pixel and saidcorresponding pixel; a discrimination image forming unit for obtaining adiscrimination image by discriminating each pixel in said intensitydifference image into a pixel having an intensity difference no lessthan a standard value and a pixel having an intensity difference lessthan the standard value; and a decision unit for detecting and decidinga region having a generally wedge-shaped set of pixels in saiddiscrimination image as an obstacle region.
 2. An obstacle detectionsystem according to claim 1, wherein said detection unit decides thatthe lowermost pixel in said wedge-shaped region of said discriminationimage is either at a point of contact between said obstacle region insaid first or second image, as corresponding to said discriminationimage, and the ground being traveled by said own vehicle, or at aportion of said obstacle region and the closest to said own vehicle. 3.An obstacle detection system according to claim 1 or 2, wherein saiddetection unit decides that such one of said generally wedge-shapedregion existing generally in the scan-line direction of said first andsecond images, as corresponding to said discrimination image, that itsside is at a higher location of said first and second images than theapexes opposed thereto, is the obstacle region.
 4. An obstacle detectionsystem according to claim 1, wherein said detection unit decides onepair of wedge-shaped regions generally of the same shape, as located ata spacing on the generally identical scan-line in said discriminationimage, and decides the region between said paired wedge-shaped regionsas the obstacle.
 5. An obstacle detection method comprising: a firstimage pickup step and a second image pickup step of obtaining a firstimage information of a first image and a second image information of asecond image, respectively, by taking the surrounding region of adriver's own vehicle substantially simultaneously as images formed of aset of pixels from light receiving units arranged at a spacing on theown vehicle; an image information storage step of storing said firstimage information and said second image information; an intensitydifference image forming step of forming an intensity difference imageby determining the corresponding pixels in said second image of thesecond image information, as assuming that an arbitrary pixel of saidfirst image of said first image information stored in said imageinformation storage step exists on the ground plane being traveled bysaid own vehicle, to determine the intensity difference between saidarbitrary pixel and said corresponding pixel; a discrimination imageforming step of obtaining a discrimination image by discriminating eachpixel in said intensity difference image into a pixel having anintensity difference no less than a standard value and a pixel having anintensity difference less than the standard value; and a decision stepof detecting and deciding a region having a generally wedge-shaped setof pixels in said discrimination image as an obstacle region.
 6. Anobstacle detection method according to claim 5, wherein said detectionstep decides that the lowermost pixel in said wedge-shaped region ofsaid discrimination image is either at a point of contact between saidobstacle region in said first or second image, as corresponding to saiddiscrimination image, and the ground being traveled by said own vehicle,or at a portion of said obstacle region and the closest to said ownvehicle.
 7. An obstacle detection method according to claim 5 or 6,wherein said detection step decides that such one of said generallywedge-shaped region existing generally in the scan-line direction ofsaid first and second images, as corresponding to said discriminationimage, that its side is at a higher location of said first and secondimages than the apexes opposed thereto, is the obstacle region.
 8. Anobstacle detection method according to claim 5, wherein said detectionstep decides one pair of wedge-shaped regions generally of the sameshape, as located at a spacing on the generally identical scan-line insaid discrimination image, and decides the region between said pairedwedge-shaped regions as the obstacle.